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INTRODUCTION 

Background 

In  November,  1973,  a  research  agreement  with  the  title 
"Evaluation  of  PLATO  IV  System  in  Air  Force  Medical  Training" 
was  signed  between  representatives  of  the  USAF  School  of  Health 
Care  Sciences  (SHCS)  at  Sheppard  AFB,  the  Air  Force  Human 
Resources  Laboratory  (AFHRL),  and  the  Defense  Advanced  Research 
Projects  Agency  (ARPA)  of  the  Department  of  Defense.  The 
agreement  called  for  the  research  and  development  of  a  new 
problem-oriented  medical  curriculum  (POMC)  for  the  Physician 
Assistant  program  offered  at  Sheppard ' s  SHCS.  The  new  curric¬ 
ulum  was  to  be  developed  and  implemented  on  the  PLATO  IV 
computer-based  education  system.  A  major  objective  of  the 
project  was  to  be  a  comparative  evaluation  of  the  instruc¬ 
tional  effectiveness  of  the  new  problem-oriented  materials 
and  the  existing  traditional  curriculum. 

Under  the  terms  of  the  agreement,  the  Air  Force's  Air 
Training  Command  (ATC)  was  to  furnish  (through  SHCS)  a  staff 
of  subject  matter  specialists  and  support  personnel  to  develop 
the  new  curriculum,  sufficient  classroom  and  laboratory  space 
to  house  the  project,  and  enough  physician  assistant  students 
to  form  the  evaluation  sample. 

AFHRL  was  to  develop  an  evaluation  plan  in  conjunction 
with  SHCS  and  to  provide  direction  for  and  monitoring  of  the 
evaluation  itself.  In  addition,  AFHRL  was  to  derive  and 
implement  measures  of  student  and  instructor  attitudes  as 
well  as  tests  of  student  performance. 

It  was  to  be  ARPA's  responsibility  to  provide  20  PLATO 
IV  terminals  plus  support  costs  for  computer  usage,  communi¬ 
cations  lines,  and  terminal  maintenance  for  three  years  of 
service  at  the  Sheppard  site.  As  well  as  providing  hardware 
support,  ARPA  was  to  fund  the  Military  Training  Centers  group 
(MTC)  of  the  University  of  Illinois'  Computer-based  Education 
Research  Laboratory  (CERL)  to  provide  SHCS  staff  with  train- 
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ing  and  advice  in  the  use  of  the  PLATO  system.  The  MTC  group 
was  also  to  cooperate  in  the  development  of  curriculum  mater¬ 
ials  for  the  project  and  perform  other  assistance  as  agreed 
to  by  the  Sheppard  project  manager. 

In  mid-197S,  at  the  time  the  Sheppard  project  had  reached 
its  implementation  stage,  the  project  sponsor  (ARPA)  asked 
CERL  to  prepare  a  proposal  to  do  an  evaluation  of  the  Shep¬ 
pard  effort  as  well  as  those  of  other  ARP A-suppor ted  PLATO 
projects.  The  MTC  group  had  taken  an  important  role  in  sup¬ 
port  of  each  of  the  major  ARPA  sites.  This  role  included 
giving  initial  training  in  the  TUTOR  language,  acting  as 
liaison  with  the  rest  of  CERL,  making  site  visits  for  in- 
service  training  and  problem  solving,  reviewing  lessons  to 
improve  their  pedagogical  approach,  etc.  Since  they  had 
observed  these  projects  from  their  inception,  the  MTC  group 
was  thought  by  ARPA  to  be  in  a  good  position  to  assist  in 
formative  evaluation  for  the  purpose  of  maximizing  the  effec¬ 
tiveness  of  each  site.  CERL's  PLATO  Educational  Evaluation 
and  Research  group  (PEER)  was  also  enlisted  to  participate  in 
the  evaluation. 

Another  goal  of  the  MTC/PEER  evaluation  which  resulted 
from  the  contract  with  ARPA  was  to  identify  generalizable 
aspects  of  the  experience  of  the  ARPA- supported  sites.  The 
evaluation  sought  to  address  the  overall  question  of  "under 
what  conditions  can  CRE  make  a  cost  effective  contribution 
to  the  instructional  needs  of  the  services".  PLATO  implementa¬ 
tions  at  Chanute  AFB  and  Aberdeen  Proving  Oround  sites  were 
studied  along  with  that  at  the  Sheppard  AFP  site.  Reports  by 
Himwich  (  1  977b),  Call-Himwich  (  1977b),  and  Avner  &  Misselt. 
(1977)  focus  on  the  experience  at  Aberdeen  Proving  Ground. 

The  Chanute  project  is  discussed  in  reports  bv  Klecka  (lQ77a; 
1977b;  1978),  Himwich  (1977a),  Misselt,  Himwich,  Francis, 

Avner,  Tatsuoka  h  Klecka  (1977),  Tatsuoka,  Misselt  &  Maritz 
(1978),  and  Tatsuoka  (1978).  The  report  by  Call-Himwich 
(1978)  on  the  POMC  courseware  is  a  companion  to  this  account 
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of  the  Sheppard  project.  Other  reports  dealing  with  more 
general  topics  include  a  study  of  critical  decisions  made 
during  various  pLATO  development  projects  (Steinberg  et  al  .  , 
1°77)  and  a  set  of  guidelines  for  site  implementation  and 
management  (Francis,  1977). 

Scope  and  Rationale 

The  Sheppard  AFB  CDF  project  had  two  distinct  phases, 
each  characterized  by  its  own  goals,  timelines,  and  organiza¬ 
tional  structures.  The  first  phase  sought  to  create  and  imple¬ 
ment  CPF  materials  for  a  problem-oriented  curriculum  in  the 
SHCS  training  program  for  phvsician  assistants.  It  lasted 
from  November ,  1973,  when  the  original  research  agreement 
was  approved,  until  December,  1975,  when  the  staff  terminated 
the  effort  due  to  attrition  in  the  student  sample  and  other 
problems  which  threatened  the  integrity  of  the  evaluation. 

After  abandoning  the  first  set  of  goals,  the  staff  proposed  a 
Collow-on  investigation  which  was  to  involve  preparation  and 
testing  of  CPF  materials  for  the  Medical  Lab,  Radiology,  and 
Dental  Assistant  courses  in  the  SHCS.  This  second  proposal 
was  approved  in  time  to  begin  development  of  materials  in 
March,  1976.  Final  data  collection  for  the  second  phase 
ended  on  June  30,  1977. 

This  report  deals  exclusively  with  the  plans,  transac¬ 
tions,  and  outcomes  of  the  first  phase  of  the  SHCS  project. 

By  virtue  of  the  fact  that  the  original  goals  of  the  project 
were  abandoned  (signifying  the  end  of  a  first  phase  and  the 
necessity  of  a  second),  the  project  failed  to  achieve  its 
Phase  I  objectives.  The  main  purpose  of  this  report  is  to 
examine  the  underlying  causes  of  that  failure  and  to  document 
them  as  lessons  learned  and  situations  to  avoid  in  future 
projects.  It  would,  of  course,  be  desirable  also  to  examine 
the  major  events  and  findings  of  the  second  phase.  Unfortun¬ 
ately,  the  late  completion  date  and  CFRL’s  reduced  level  of 


contact  with  Shepp<r~d  during  1Q76— 1077  make  it  impossible  to 
extend  the  scone  of  our  evaluation  beyond  Phase  I.  A  full 
report  of  the  Phase  II  effort  is  to  be  made  by  AFHRL  after 
data  analysis  and  interpretation  are  completed. 

The  information  on  which  the  present  report  is  based  was 
gathered  informally  throughout  CER1. 's  association  with  the 
Sheppard  project.  Correspondence,  documents  produced  by  the 
Sheppard  staff,  notes  made  by  MTr  consultants  during  site 
visits,  and  interviews  with  staff  members  are  the  major  sources 
of  data  we  have  had  available.  The  report  by  project  staff 
and  AFHRL  researchers  ( Ste i n ker chner ,  Deignan,  Waters  ^ 

DeLeo,  1977  )  has  also  provided  useful  insight,  and  we  are 
grateful  to  AFHRL  for  giving  us  access  to  a  prepublication 
copy.  Although  we  have  utilized  numerous  sources  of  informa¬ 
tion  to  prenare  this  document,  we  recognize  that  other  forms 
of  data  (e.g.,  interviews  with  course  instructors,  and  closer 
observation  of  implementation  conditions)  would  have  signifi¬ 
cantly  enhanced  our  understanding  of  Phase  I. 

Much  can  be  learned  from  Sheppard's  Phase  I  effort  in 
spite  of  its  failure  to  fully  implement  and  evaluate  the 
problem-oriented  approach  to  medical  education.  In  order  to 
focus  on  the  major  outcomes  of  Phase  I,  some  topics  (e.g., 
the  management  of  the  courseware  development  and  implementa¬ 
tion  activities  and  the  ways  in  which  CFRL  contributed  to  them) 
are  discussed  only  briefly.  However,  much  of  this  information 
is  reDorted  elsewhere  (see  Steinberg  et  al . ,  1^77:  Francis, 

1 077;  and  Steinkerchner  et  al  .  ,  1977). 

The  report  is  organized  in  four  major  sections.  Follow¬ 
ing  the  introduction,  a  brief  history  of  major  events  is  given. 
The  subsequent  section  identifies  the  four  most  significant 
outcomes  or  problem  areas  and  analyzes  their  causes.  The 
final  section  gives  a  summary  of  the  major  outcomes,  briefly 
states  some  additional  outcomes,  and  gives  an  overall 
assessment  of  the  Phase  I  effort. 
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HISTORICAL  SUMMARY 


Project  Origins 

The  Request  for  Training  Research  (RTR)  dated  4  April 
1°73  is  the  earliest  official  document  generated  during  the 
Sheppard  project  (see  Appendix  A).  Prepared  by  an  official 
in  the  School  of  Health  Care  Sciences  (hereafter  known  as 
the  "project  initiator"),  the  RTR  outlined  a  rationale  for 
the  research  effort  and  set  out  general  plans  for  what  was 
to  be  attempted . 

In  essence,  the  RTR  described  a  shortage  of  physicians' 
services  in  the  IJ .  S.  Air  Force  which  could  potentially  be 
alleviated  through  the  training  of  large  numbers  of  physician 
assistants  (PAs).  A  training  program  for  PAs  had  been  estab¬ 
lished  at  the  SHCS  in  about  1970  with  a  comprehensive  Plan  of 
Instruction  being  completed  in  August,  1972.  In  the  view  of 
the  project  initiator,  the  existing  PA  curriculum  suffered 
from  the  lack  of  a  basic  frame  of  reference  for  the  core  of 
medical  knowledge  to  be  memorized.  He  proposed  a  curricular 
reorganization  intended  to  illustrate  the  clinical  relevance 
of  this  material.  Although  it  is  not  stated  explicitly,  the 
reader  of  the  RTR  may  infer  that  a  main  benefit  expected  from 
increased  clinical  relevance  was  a  decrease  in  training  time. 
Part  of  the  impetus  for  a  reworking  of  the  PA  curriculum  was 
the  frequency  of  student  questions  about  why  a  PA  would  need 
to  know  certain  basic  science  material.  Hence,  making  the 
curriculum  more  clinically  relevant  was  expected  to  improve 
student  attitudes.  Another  supposed  benefit  was  that  other 
allied  health  personnel  would  gain  a  better  understanding  of 
their  own  place  along  the  "diagnosis-treatment"  loop. 

Hopes  for  redeveloping  the  PA  course  as  a  problem- 
oriented  medical  curriculum  actually  preceded  the  plans  to 
use  the  PLATO  system  in  this  effort.  It  is  our  understanding 
that  officials  in  the  SHCS  learned  in  early  1973  that  a  set 
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of  20  PLATO  IV  terminals  originally  intended  by  ARPA  for  one 
of  the  other  services  might  be  available  for  Air  Force  use. 

The  project  initiator  saw  this  as  an  opportunity  to  fund  the 
initial  work  on  the  POMC:  he  apparently  was  more  interested 
in  developing  the  new  curriculum  than  in  doing  research  with 
computer  based  education  per  se.  The  initial  plans  did, 
however,  leave  open  the  possibility  that  "non-POMC"  CRE 
research  might  be  attempted,  as  is  indicated  by  the  following 
statement  from  the  RTR: 

During  the  development  of  this  project,  students 
will  be  exposed  to  both  curricular  formats  (POMC 
and  traditional)  in  which  PLATO  TV  may  or  may  not 
be  employed.  The  suitability  and  contribution  of 
computer-assi sted  instruction  will  be  evaluated  as 
it  stands  alone  and  vis-a-vis  the  curricula. 

During  the  remainder  of  the  spring  and  summer  of  1°73 
there  were  a  number  of  meetings  between  representatives  of 
the  SHCS  staff  and  those  of  other  agencies  concerned  with 
the  project  (CERL.,  ATC,  and  AFHRL)  in  order  further  to  refine 
the  plans.  Because  of  the  relative  newness  of  the  PLATO  IV 
system  and  general  lack  of  experience  with  large  CBE  curric¬ 
ulum  development  projects,  ARPA  agreed  to  fund  CFRL  to  provide 
support  personnel  for  the  Sheppard  effort.  Another  event 
during  this  period  was  the  attempt  by  the  project  initiator 
and  other  MDs  on  the  PA  course  staff  to  draw  up  a  Dlan  to 
organize  the  PA  curriculum  around  2^  sets  of  symptoms  or 
"problems".  Apparently  it  was  still  hoped  that  a  full  reor¬ 
ganization  of  the  curriculum  could  be  done,  with  C^F  to  be 
applied  only  to  a  part  of  it. 

Project  Organization 

By  November,  1973,  a  formal  research  agreement  had  been 
signed  and  the  project  was  officially  underway.  It  was  also 
about  this  time  that  the  project  initiator  finalized  his  plans 
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to  leave  the  Air  Force  and  an  experienced  administrator  was 
selected  to  become  acting  director.  In  December,  1973,  the 
acting  director,  the  project  initiator,  and  two  other  SHCS 
officials  participated  in  planning  discussions  with  members 
of  CERL's  MTC  support  group. 

During  the  remainder  of  1973  and  the  first  few  weeks  of 
1979,  a  number  of  organizational  and  staffing  decisions  were 
made.  The  acting  director,  having  taken  up  residency  at 
Sheppard  AFB  during  this  period,  arranged  for  office  space  to 
house  the  project  and  its  PLATO  IV  terminals.  He  also  selected 
a  number  of  staff  members,  some  of  whom  were  to  join  the 
project  in  February  with  others  not  coming  until  June  or 
July,  1979. 

A  series  of  planning  meetings  was  held  at  the  SHCS  in 
late  February.  Representatives  of  the  MTC  group  met  together 
with  personnel  from  Sheppard's  PLATO  IV  staff  and  instructors 
from  the  Physician  Assistant's  course.  Among  the  major  out¬ 
comes  of  these  meetings  were:  (a)  a  working  definition  of 
a  problem-oriented  medical  curriculum,  (b)  an  understanding 
that  the  PLATO  project  would  constitute  the  core  of  the 
research  on  the  PA  course  (and  that  it  was  beyond  the  scope 
of  the  project  to  undertake  the  revision  of  the  PA  curriculum 
as  a  whole),  (c)  an  indication  from  the  project  staff  that 
the  evaluation  of  PLATO  IV  as  an  instructional  medium  would 
be  secondary  to  the  evaluation  of  the  POMC,  (d)  disclosure  of 
plans  to  divide  the  development  effort  into  a  basic  science 
component  and  a  problem-oriented  component,  (e)  a  description 
of  the  intent  to  use  PLATO  in  a  wide  variety  of  specific  prob¬ 
lem  areas  within  the  course,  and  ( f )  discussion  of  the  role 
that  might  be  taken  by  CERL's  MTC  group. 

While  awaiting  the  arrival  of  the  primary  subject  matter 
specialist  (a  newly-graduated  physician),  the  staff  members 
already  "on  board"  traveled  to  CERL  to  receive  training  in 
the  TUTOR  language,  instructional  design,  and  other  aspects 
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of  the  use  of  PLATO  IV  (see  Francis,  1976).  Upon  return  to 
Sheppard  AFB,  they  continued  to  develop  their  skills  in  lesson 
writing  while  refining  the  plans  for  the  new  curriculum. 

Work  on  a  number  of  basic  science  lessons  was  begun  while 
awaiting  the  physician,  the  "clinical"  subject  matter  expert. 

The  weeks  following  the  arrival  of  the  physician  were 
marked  by  a  significant  alteration  of  the  project's  direc¬ 
tion.  Whereas  it  had  previously  been  planned  to  develop  CBE 
materials  for  each  of  a  number  of  topics  which  were  thought 
to  be  well-suited  for  CRE  application,  it  was  decided  during 
this  period  that  all  of  the  material  related  to  a  single 
organ  system  (the  respiratory  system)  should  be  targeted  for 
the  POMC.  The  portions  of  the  course  Plan  of  Instruction 
(POI)  dealing  with  respiration  w°re  identified  and  these 
tonics  were  reorganized  and  reordered  for  incorporation  into 
the  POMC.  Concurrent  with  the  choice  of  respiration  as  sub¬ 
ject  matter  was  a  reconsideration/redefinition  of  the  POMC's 
nature.  The  subject  matter  specialist  was  given  primary 
responsibility  for  planning  the  curriculum,  and  he  reinter¬ 
preted  the  definition  in  a  manner  with  which  he  could  feel 
comfortable.  A  fuller  discussion  of  this  point  is  given 
later  in  the  report. 

Initial  develonment.  work  on  elements  of  both  the  basic 
science  component  and  a  clinically-oriented  portion  of  the 
POMC  proceeded  through  the  summer  and  fall  of  1Q74.  Among 
the  major  events  of  this  period  were:  (a)  the  arrival  of 
the  remaining  staff  members  allocated  to  the  project  (the 
last  of  whom  was  not  assigned  until  November--one  year  after 
the  project  got  underway),  and  (b)  the  initial  planning  for 
the  implementation  and  evaluation  phase. 

Planning  for  the  Implementation 

Since  plans  for  redeveloping  the  entire  PA  curriculum 
to  be  problem-oriented  had  been  abandoned  or  postponed,  the 
implementation  of  the  POMC  had  to  be  done  in  such  a  way  as 
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to  minimize  disruption  of  the  regular  course.  An  implementa¬ 
tion  plan  was  drawn  up  (see  Appendix  B)  which  called  for  the 
replacement  of  discrete  hours  of  the  regular  curriculum  with 
new  CBE  materials.  In  order  to  provide  for  a  comparative 
evaluation,  the  project  staff  planned  to  select  a  sample  of 
16  students  for  the  experimental  (POMC)  group  with  the  remain¬ 
ing  48  students  assigned  to  the  regular  curriculum.  Only  16 
subjects  were  to  be  assigned  to  the  experimental  condition 
because  there  were  only  to  be  16  PLATO  IV  terminals  for  stu¬ 
dent  use,  and  it  was  believed  to  be  necessary  to  have  a 
student  to  terminal  ratio  of  1  to  1 .  To  increase  the  size  of 
the  n,  however,  the  implementation  plan  provided  for  a 
replication  of  the  comparison  with  the  use  of  a  second  class 
of  PA  students.  Flence,  a  total  of  32  students  were  to  be 
exposed  to  the  POMC  during  the  evaluation  phase.  An  important 
consequence  of  the  decision  to  use  a  second  class  of  PA  train¬ 
ees  was  that  the  start  of  the  implementation  had  to  be  moved 
forward  from  January,  1976,  to  October,  1975,  and  finally  to 
July,  1975.  Hence,  the  time  available  for  lesson  development 
was  shortened  by  six  months. 

An  evaluation  plan  was  prepared  in  March,  1975,  to  out¬ 
line  specifications  for  data  collection  and  analysis  during 
the  comparison  (see  Appendix  C).  It  called  for  the  use  of  a 
comprehensive  examination  over  the  respiratory  topics  covered 
in  both  curricula.  The  two  groups  were  also  to  be  compared 
on  their  performance  in  a  set  of  simulated  patient  encounters. 
These  latter  exercises  were  expected  to  measure  the  problem¬ 
solving  abilities  of  the  PA  trainees  in  the  task  of  diagnosis 
and  treatment  of  respiratory  diseases  and  other  patient  ail¬ 
ments.  The  measurement  of  student  attitudes  toward  the  POMC 
and  toward  computer-based  education  was  also  an  important 
component  of  the  evaluation.  The  plan  called  for  two  adminis¬ 
trations  of  an  attitude  questionnaire--one  after  a  brief 
exposure  to  the  POMC  and  the  other  after  completion  of  the 
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first  year  of  the  PA  program.  The  reliability  and  incorpor- 
ability  of  the  PLATO  system  were  also  to  be  investigated. 

Results  of  the  Implementation 

In  spite  of  the  fact  that  some  of  the  CBF  lessons  slated 
for  first  trimester  use  were  not  yet  ready,  the  implementation 
got  underway  in  July,  1975,  as  scheduled.  The  first  class's 
initial  use  of  the  C9E  lessons  did  not  go  smoothly.  Students 
encountered  a  number  of  coding  errors  in  the  lessons,  and 
there  was  an  unfortunate  mixup  among  the  project  staff  as  to 
the  policies  on  student  dress  and  behavior  in  the  PLATO  labora¬ 
tory.  The  students  were  first  told  by  some  staff  members 
that  an  informal  atmosphere  would  be  maintained.  This  "policy" 
was  quickly  rescinded  by  the  project  director,  who  felt  that 
frequent  visits  from  Air  Force  "brass"  would  require  conven¬ 
tional  dress  and  attendance  patterns. 

Scheduling  of  PLATO  IV  study  sessions  was  a  difficult 
problem.  The  16  students  in  the  experimental  group  were 
excused  from  the  regular  class  sessions  in  which  respiratory 
topics  were  to  be  discussed  in  order  that  they  might  receive 
alternative  instruction  on  the  PLATO  IV  system.  Since  it  was 
necessary  for  them  to  rejoin  the  regular  class  at  the  next 
( nonrespiratory)  sessions,  they  sometimes  were  not  able  to 
complete  the  assigned  CBF  lessons  in  the  available  time.  To 
make  up  for  such  lost  study  time  and  to  have  opportunity  for 
review  of  CBF  materials,  the  POMC  students  often  had  to  return 
to  the  PLATO  IV  classroom  during  evening  hours. 

The  late  summer  months  of  1975  were  very  busy  ones  for 
the  project  personnel.  The  severe  time  pressure  on  the  staff 
to  commence  student  use  in  July  (which  had  left  too  little 
time  for  "debugging"  of  programs  and  coordination  of  policies 
regarding  student  dress  and  behavior)  continued  through  the 
summer  and  fall.  Not  only  did  the  staff  have  to  rush  to 
complete  the  first  trimester  lessons  for  the  initial  group 


of  students,  they  also  had  to  prepare  for  the  start  of  the 
second  trimester.  A  number  of  second  trimester  lessons 
remained  to  be  completed  ,  and  the  first  trimester  lessons 
required  revisions  prior  to  being  used  with  the  next  entering 
class.  It  was  only  through  considerable  effort  and  much 
overtime  work  that  the  project  staff  was  able  to  meet  its 
schedule.  Nevertheless,  the  second  trimester  got  underway 
with  a  second  group  of  16  students  being  assigned  to  the  POMC 
condition.  By  this  time  there  had  been  some  attrition  in  the 
first  class’s  experimental  group  so  that  only  eight  students 
continued  to  the  second  trimester's  material. 

Of  the  eight  students  from  the  first  class  who  had  left 
the  POMC  group  by  the  start  of  the  second  trimester,  five  had 
been  academically  eliminated  from  the  PA  program.  Three  others 
had  remained  in  the  program  but  had  requested  to  be  dropped 
from  the  experiment  in  order  to  return  to  the  regular  class¬ 
room.  Before  the  second  trimester  was  completed,  six  of  the 
remaining  eight  students  also  asked  to  go  back  to  the  class¬ 
room.  As  with  the  first  class,  the  second  set  of  PA  students 
reacted  negatively  to  the  experimental  conditions.  Realiz¬ 
ing  that  "nothing  further  could  be  done  to  resolve  the  miscon¬ 
ceptions,  relieve  the  anxieties,  or  change  the  negative 
attitudes  of  the  students",  the  staff  decided  it  would  be  a 
waste  of  time  and  money  to  continue  the  project  with  so  few 
students  and  voted  to  terminate  the  POMC  effort  in  December, 
1975  (Steinkerchner  et  al.,  1977). 

At  the  time  of  the  decision  to  discontinue  the  POMC, 
most  of  the  CBE  lessons  planned  for  the  first  and  second 
trimesters  had  been  completed.  Many  of  the  third  trimester 
materials,  however,  had  not.  Some  of  these  latter  lessons 
were  to  take  the  form  of  simulated  patient  encounters.  Since 
the  basic  format  of  the  simulations  had  been  determined 
earlier,  all  that  remained  was  to  input  the  "data"  from  a  set 
of  actual  clinical  cases.  The  remaining  third  trimester 
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lessons,  on  the  other  hand,  would  heave  required  more  effort 
to  develop.  It  is  possible,  but  not  likely,  that  thev  would 
have  been  completed  in  time  for  use  had  the  POMC  students  not 
opted  to  leave  the  experiment. 

Because  of  the  early  termination,  much  of  the  evaluation 
data  called  for  by  the  evaluation  plan  (Appendix  C)  were  not 
collected.  There  was  no  direct  comparison  of  the  POMC  and 
regular  curricula  with  regard  to  instructional  effectiveness. 
Neither  the  knowledge  of  basic  science  material  nor  clinical 
problem-solving  skills  were  compared  between  the  two  groups. 

The  attitudinal  data  were  collected,  but  the  second  adminis¬ 
tration  of  the  questionnaire  came  at  the  time  of  the  discon¬ 
tinuance  of  the  POMC  rather  than  at  the  end  of  the  third  tri¬ 
mester  as  had  originally  been  planned.  Unfortunately,  the 
basic  questions  regarding  the  value  of  a  problem-oriented 
curriculum  in  physician  assistant  training  were  not  answered 
by  the  data  collected  during  the  Sheppard  project. 

Pegrouning  for-  Pha_se  T_I 

At  about  the  time  that  the  Sheppard  staff  decided  to 
close  out  experimentation  with  the  POMC ,  ARPA  and  CERI.  came 
to  an  agreement  that  the  MTC  group's  support  role  would  be 
curtailed.  Although  the  MTC  group  had  little  opportunity 
under  its  changed  role  to  keep  abreast  of  the  planning  and 
implementation  of  Sheppard’s  second  phase  effort,  it  is 
important  to  indicate  something  of  the  nature  of  the  second  phase. 

Planning  for  the  follow-on  project  (or  second  phase)  Imran 
in  December,  197S.  Looking  for  courses  in  which  there  was  a 
high  student  flow  and  in  which  a  high  level  of  faculty  involve¬ 
ment  was  likely,  the  Sheppard  staff  selected  the  Medical  I. ah, 
Radiologv,  and  Dental  Assistant  courses  within  the  SHCS. 

Other  planning  considerations  were:  (a)  the  requirement  that 
identical  criterion  measures  be  administered  to  CRF  and  con¬ 
trol  groups  at  the  lesson  test  and  block  test  levels,  (h) 
thorough  measurement  of  learners'  entering  characteristics, 
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and  (c)  the  capability  of  running  separate  CBF  and  control 
groups  for  an  entire  block  of  instruction  ( Steinkerchner  et 
al  .  ,  1977).  Tbe  criterion  of  high  instructor  involvement 
demonstrates  that  the  staff  wanted  to  identify  courses  whose 
instructors  felt  thev  had  problems  that  could  be  solved  by 
use  of  PLATO.  The  PLATO  staff  had  learned  the  importance  of 
high  instructor  involvement  from  their  experience  in  the  POMC 
phase  of  the  project. 

Development  work  on  the  new  materials  began  in  March, 
197b,  with  preliminary  data  collection  scheduled  to  take 
place  later  that  summer.  Peoause  the  original  contract  with 
ARPA  was  to  expire  September  30,  1976,  an  extension  was  funded 
until  June  30,  1977.  Formal  data  collection  for  a  summative 
evaluation  was  to  commence  in  November,  1976,  and  continue 
until  the  end  of  the  ARPA-funded  extension.  A  final  report 
was  scheduled  for  completion  by  December,  1977. 

According  to  the  research  agreement  for  the  extension 
period  (see  Appendix  D),  the  main  goals  of  the  follow-on 
effort  and  its  evaluation  were:  (a)  comparison  of  instruc¬ 
tional  effectiveness  of  CBF  lessons  and  lessons  presented 
by  lecture  or  by  conventional  self-Daced  media,  (b)  determina¬ 
tion  of  costs  of  CBF  and  non-CBF  instruction  in  the  three 
courses,  (c)  determination  of  factors  underlying  student 
attitudes  toward  CBE ,  and  (d)  determination  of  factors  under¬ 
lying  attitudes  of  instructors  toward  CBF. 
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ANALYSIS  OF  PROJFCT  OUTCOMES 

Student  Wi thd rawal  from  Ex  per  imen t 

The  major  reason  for  the  staff’s  decision  to  abandon  the 
POMC  effort  was  the  heavy  loss  of  subjects  from  the  experi¬ 
mental  group  due  to  academic  failure  and  dissatisfaction  with 
the  experimental  conditions.  As  stated  above,  five  of  the 
original  16  POMC  students  ( 3 1  % )  were  eliminated  from  the  PA 
program  due  to  poor  academic  performance.  Only  six  of  the  48 
control-group  students  (12.5%)  were  dropped  during  the  same 
period.  In  spite  of  its  higher  dropout  rate,  the  POMC  group's 
five  eliminees  did  not  feel  that  their  failure  was  related  to 
participation  in  the  experiment.  Instead,  they  attributed 
their  problems  to  rapid  pacing  in  the  regular  portion  of  the 
course  ( Ste inkerchner  et  al  .  ,  1977).  Nevertheless,  the 
losses  due  to  academic  attrition  dealt  a  severe  blow  to  the 
project’s  chances  for  a  successful  outcome. 

Although  the  academic  eliminations  may  not  have  been 
related  to  the  POMC  or  its  implementation,  eight  of  the 
remaining  eleven  students  in  the  first  class  asked  to  leave 
the  POMC  group  because  they  were  unhappy  with  the  experimental 
conditions.  This  mass  d issa t i sf ac t ion  was  expressed  informally 
at  first  through  individual  complaints  to  the  Sheppard  PLATO 
staff.  Ultimately,  however,  the  dissident  group  submitted 
formal  letters  of  request  that  they  be  transferred  bock  to 
the  regular  classroom.  This  latter  action  prompted  the 
project  staff  to  terminate  the  POMC  effort. 

The  underlying  causes  of  student  dissatisfaction--perhaps 
the  most  striking  outcome  of  the  POMC  effort--are  numerous, 
but  they  are  rooted  in  the  severe  implementation  constraints 
imposed  on  the  project.  A  discussion  of  these  constraints 
must  begin  with  an  understanding  of  the  students'  perspective. 

As  Physician  Assistant  trainees,  the  students  wer.  a 
h i phi y-mot ivated  and  strongly  goal-oriented  group.  They  had 
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all  worked  in  health  care  specialties  during  their  Air  Force 
or  flavy  careers  and  many  were  hopeful  of  working  as  PAs  after 
leaving  the  service.  Career  aspirations  and  family  pressures 
made  success  in  the  PA  curriculum  a  very  important  objective. 

Because  nearly  all  of  the  academic  training  was  covered 
in  the  first  year  of  the  two  year  program,  the  general  pace 
of  the  course  was  rapid  and  tightly  scheduled.  Students  spent 
six  to  eight  hours  in  classwork  each  day  and  were  expected 
to  complete  reading  assignments  and  outside  study  during  the 
evening  hours  and  on  weekends.  Most  participated  in  small 
study  groups  to  review  lecture  notes  and  prepare  for  the 
frequent  examinations  in  the  various  subject  areas.  Hence, 
there  was  ample  opportunity  for  them  to  exchange  views  on 
instructors,  events  of  the  day,  and  their  progress  through 
the  curriculum,  as  well  as  to  discuss  specific  items  of 
course  content. 

The  POMC  implementation  plan  (see  Aopendix  B)  called  for 
the  replacement  of  discrete  hours  of  classroom  lecture  with 
newly-developed  CBF  lessons.  The  16  students  selected  for 
the  POMC  group  (one  fourth  of  the  class  of  64)  were  to  leave 
the  classroom  during  the  times  the  target  respiratory  materials 
were  to  be  presented  in  order  to  study  newly-created  lessons 
from  the  POMC.  Since  the  POMC  involved  a  reorganization  of 
the  respiratory  content  of  the  curriculum,  the  topic  studied 
by  the  experimental  group  on  a  particular  day  was  often  quite 
different  from  that  covered  in  the  regular  classroom. 

The  PLATO  staff  tried  to  assure  the  experimental  group 
students  that  although  the  POMC  would  present  topics  in  a 
different  order,  eventually  all  would  be  covered.  Nevertheless, 
the  students  were  frequently  disconcerted  to  find  that  the 
classroom  instructor  had  presented  "new"  material  that  they 
had  "missed"  while  attending  a  PLATO  session.  Because  of 
their  compulsion  to  do  well  in  the  course,  these  students 
often  felt  that  they  had  to  work  harder  to  "keep  up"  with  the 
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class.  This  presumably  meant  that  they  spent  time  studying 
the  lecture  notes  of  friends  or  in  doing  "extra"  reading 
assignments.  The  concern  about  missing  lectures  was  so  great 
that  some  students  considered  the  POMC  to  be  an  additional 
course  rather  than  an  alternative  presentation  of  some  of  the 
same  mater i al . 

We  have  not  had  access  to  the  letters  of  "mutiny",  but 
it  is  apparent  from  the  interview  and  questionnaire  data 
summarized  by  Steinkerchner  et  al .  (1977)  and  the  informal 
reports  of  project  members  that  the  major  complaint  of  the 
students  was  that  they  felt  they  were  not  getting  the  same 
information  as  that  presented  to  the  rest  of  the  class.  The 
second  most  frequent  complaint  was  directed  at  the  test  grad¬ 
ing  procedures  used  to  accommodate  the  experimental  conditions 
According  to  the  nlan,  the  regular  instructors  were  to  meet 
with  a  PLATO  staff  representative  to  discuss  each  test  prior 
to  its  administration.  If  the  test  contained  items  that  had 
not  yet  been  taught  in  the  POMC,  the  instructor  was  to  identi¬ 
fy  them  as  items  to  he  omitted  bv  the  PO”C  students.  The 
PLATO  staff,  on  the  other  hand,  was  to  be  responsible  for 
testing  the  POMC  students  over  those  items  at  a  later  date. 

The  test  scores  for  the  POMC  group  were  to  be  recorded  as  the 
percentage  of  "non-omitted"  items  that  were  answered  correctly 
Scores  were  later  adjusted  to  reflect  performance  on  the 
make-up  tests  given  by  the  PLATO  staff.  The  procedure  was 
complicated  at  best  and  it  was  not  implemented  smooth1 v  at 
least  one  instructor  in  the  course  was  never  told  how  it  was 
to  work.  Students  distrusted  the  process  and,  because  or 
their  great  concern  with  course  grades,  they  were  often  upset 
at  not  being  given  credit  for  correct  responses  to  "omitted" 

items . 

Another  student  complaint  was  that  they  often  lacked  a 
sense  of  where  they  were  going  in  the  POMC,  i.e.,  they  aid 
not  have  a  concept  of  the  overall  organization  of  the  new 
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curriculum.  The  uncertain  rate  of  development  of  future  les¬ 
sons  made  it  difficult  to  publish  a  comnrehonsi ve  schedule 
or  syllabus  for  student  use.  Furthermore,  the  scattered 
distribution  of  respiratory  topics  in  the  regular  Plan  of 
Instruction,  and  hence  the  sporadic  scheduling  of  PLATO 
sessions,  made  the  ?0MO  appear  especially  disjointed  and 
fragmented.  Recause  of  the  lengthy  intervals  between  PLATO 
sessions,  students  following  a  tightly-knit  argument  or  pre¬ 
sentation  in  the  POMC  sequence  often  found  it  necessary  to 
review  previous  lessons  before  continuing  through  the  curric- 
11 1  urn . 

As  mentioned  earlier,  the  POMC  students  ultimately 
stated  that  contact  with  the  PLATO  staff  was  among  the  most 
positive  aspects  of  their  experience,  but  they  may  not  have 
hern  favorably  impressed  at  the  outset.  They  were  told  in 
their  initial  orientation  to  the  project  that  they  would  have 
certain  freedoms  in  dress,  attendance  patterns,  and  behavior 
while  in  the  PLATO  classroom.  These  raised  expectations 
were  soon  lowered  attain  when  the  project  director  returned 
from  an  out-of-town  assignment.  Realizing  that  the  project 
would  have  high  visibility  (particularly  amonm  high  ranking 
visitors  to  the  SHC.S) ,  he  rescinded  the  promised  freedoms  and 
insisted  instead  on  a  high  standard  of  military  decorum.  The 
impact  of  this  mixup  in  communication  among  the  staff  (i.e., 
failing  to  settle  on  a  consistent  policy  for  student  dress 
and  attendance  prior  to  the  initial  orientation)  is  difficult 
to  gauge.  Doubtless,  however,  it  did  not  leave  a  favorable 
impression . 

Another  problem  that  dampened  students'  initial  enthusi¬ 
asm  for  the  CBF  project  was  that  the  lessons  had  not  been  com¬ 
pletely  "debugged".  There  were  still  a  number  of  coding 
errors  which  detracted  from  the  instructional  presentation. 
These  errors  and  the  policy  mixup  described  above  were  prob¬ 
ably  all  due  to  the  great  time  pressure  placed  on  the  project 
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staff'  by  the  implementation  schedule  they  adopted.  There  was 
too  little  opportunity  to  do  adequate  testing  of  the  lessons 
and  to  plan  for  a  smooth  implementation. 

There  were  also  a  number  of  complaints  about  lesson  qual¬ 
ity.  In  reviewing  the  discussion  by  Steinkerchner  et  al .  (1077), 
one  may  form  the  impression  that  many  of  these  complaints 
were  due  to  an  abnormally  high  number  of  communication  line 
errors  (a  troublesome  but  solvable  hardware  problem)  rather 
than  the  instructional  content  of  the  lessons.  Wot  all  of 
the  lesson  quality  complaints  can  be  so  easily  dismissed, 
however,  because  the  quality  control  procedures  were  not  rig¬ 
orous  and  were  not  systematically  enforced.  To  a  large  extent, 
each  author  was  allowed  to  make  his  own  judgments  as  to  when 
lessons  were  ready  for  student  use  in  the  curriculum.  Some 
lessons  were  tested  on  members  of  the  general  target  popula¬ 
tion,  but  many  were  not.  Nothing  like  the  validation  proce¬ 
dure  suggested  in  the  Air  Force  Instructional  Systems  Design 
manuals  was  applied  to  the  Sheppard  lessons.  It  might  well 
he  expected,  therefore,  that  there  wou]d  be  some  pedagogical 
deficiencies  in  these  basically  "untested"  materials.  Never¬ 
theless,  it  is  apparent  that  perceived  shortcomings  in  lesson 
qualitv  were  only  a  secondary  source  of  dissatisfaction.  The 
major  problems  seen  by  the  students  had  to  do  with  the  nature 
of  the  experiment's  implementat  ion--not  the  quality  of  the 
P0MC  lessons. 

Most  of  these  student  complaints  seem  to  stem  from  a 
feeling  of  being  in  a  minority  of  "guinea  pigs"  who  were  being 
treated  differently  than  their  peer".  Given  the  reactive 
arrangements  of  the  POMC  implementation,  it  is  not  difficult 
to  understand  why  they  may  have  held  this  feeling.  Perhans 
their  anxiety  would  have  been  less  pervasive  had  it  been 
possible  to  assign  a  larger  sample  to  the  POMC  condition. 

If  half  or  more  of  the  students  had  been  exposed  to  the  CRF 
materials,  it  is  likely  that  they  would  have  been  less  con- 


cerned  about  missed  class  sessions,  study  of  topics  in  a 
different  order,  and  uncertainties  about  curricular  organiz¬ 
ation.  Being  in  the  majority,  they  may  not  have  felt  anxiety 
that  they  were  being  treated  differently  than  (and  perhaps 
not  as  well  as)  the  students  in  the  traditional  curriculum. 

Why,  then,  were  only  one  fourth  of  the  students  assigned 
to  the  POMC?  Basically,  it  was  because  the  high  information 
content  and  rapid  pacing  of  the  PA  curriculum  led  to  rigid 
scheduling  of  student  time.  The  fixed  schedule  meant  that 
there  were  relatively  few  free  hours  during  which  the  students 
were  able  to  use  the  PLATO  terminals.  To  maximize  terminal 
availability  when  they  might  be  needed,  the  project  staff 
decided  to  assign  only  one  student  to  the  experimental  group 
for  each  student  terminal  available.  Four  of  the  20  terminals 
were  used  for  authors,  leaving  only  16  for  student  use.  Hence 
the  experimental  group  n  was  set  at  16.  It  should  be  noted 
that  the  rigid  scheduling  of  student  time--not  the  number  of 
terminal s— was  the  true  limiting  factor.  Because  the  students 
time  was  so  structured  in  the  PA  curriculum,  they  couldn't 
make  efficient  of  the  terminal  hours  that  were  available. 

Other  approaches  to  the  scheduling  and  implementation  problem 
were  considered,  but  were  judged  to  be  even  more  fraught  with 
difficulties  than  the  implementation  approach  described  here. 

In  summary,  the  POMC  phase  was  curtailed  because  of  stu¬ 
dent  dissatisfaction  with  its  implementation.  The  implementa¬ 
tion  placed  a  great  strain  on  the  experimental  group  students 
because  they  were  a  minority  and  were  treated  much  differently 
than  their  peers  in  the  regular  curriculum.  The  reactive 
arrangements  associated  with  the  POMC  "minority"  were  neces¬ 
sary  because  of  the  fundamental  constraints  on  student  time 
in  the  PA  program.  Thus  it  was  these  course  constraints 
which  were  the  basic  cause  of  POMC  effort's  downfall. 
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Incomplete  Development  of  the  PQMC 

Even  if  there  had  been  no  "mutiny"  as  the  first  class 
of  POMC  students  completed  the  second  trimester  of  their  pro¬ 
gram,  it  is  not  certain  that  the  remaining  third  trimester 
lessons  would  have  been  completed  in  time.  The  fact  that 
lesson  production  rates  fell  behind  schedule  must  therefore 
be  the  second  major  project  outcome  to  be  considered. 

The  original  research  agreement  stated  that  200  hours 
of  CBE  instruction  would  be  developed  for  the  problem-oriented 
curriculum.  Later,  at  the  time  it  was  decided  to  focus  on 
the  topic  of  respiration,  the  staff  reduced  the  total  to  145 
hours.  This  was  felt  to  be  justified  in  that  the  200  hour 
figure  had  been  arbitrarily  selected  and  because  the  smaller 
figure  represented  approx imatel y  10?>  of  the  1400  hours  com¬ 
prising  the  first  year  of  the  PA  program.  According  to  the 
implementation  plan  (see  Appendix  B) ,  the  145  Plan  of  Instruc¬ 
tion  hours  were  to  include  121  hours  of  clinical  and  basic 
science  materials  distributed  throughout  the  year  (i.e.,  24, 
25,  and  72  hours  across  the  three  trimesters,  respectively). 

In  addition,  there  were  to  be  24  hours  of  patient  encounter 
simulations  which  had  no  counterpart  in  the  regular  PA 
curriculum . 

By  the  end  of  July,  1975,  however,  there  had  been  a 
further  reduction  in  the  amount  of  materials  to  be  developed. 

A  project  document  prepared  for  ARPA  indicates  that  104  hours 
of  clinical  and  basic  science  lessons  (22,  29,  and  53  hours 
for  the  1st,  2nd,  and  3rd  trimesters,  respectively)  were 
planned.  These,  plus  the  24  hours  of  patient  encounter  simu¬ 
lations,  gave  a  projected  total  of  128  P0I  hours  of  materials. 

The  report  prepared  for  ARPA  also  gave  estimates  of  the 
actual  clock  hours  to  be  consumed  in  studying  the  CBE  materi¬ 
als.  Whereas  the  earlier  estimates  had  been  reported  in 
terms  of  P0I  hours  allotted  in  the  course  documents  for  the 


respiratory  topics,  the  project  staff  believed  that  fewer 
clock  hours  would  be  needed  for  the  CBE  lessons.  To  arrive 
at  the  estimates  of  clock  hours,  the  staff  assumed  that  251 
of  the  POI  time  could  be  saved  because  progress  through  the 
POMC  would  be  self-paced  rather  than  lock  step.  They  also 
assumed  that  an  additional  151  could  be  saved  with  CBF  as 
compared  to  other  self-paced  media.  Without  questioning  the 
basis  for  these  assumptions,  it  is  interesting  to  note  that 
the  staff  projected  11.4,  13.1,  and  24.2  clock  hours  of 
replacement  POMC  lessons  to  be  used  in  the  first,  second,  and 
third  trimesters,  respectively.  These  values  are  considerably 
smaller  than  their  POI  hour  counterparts  given  in  the  same 
document  (i.e.,  22,  29,  and  53)  or  in  the  implementation  plan 
itself  (24,  25,  and  72).  No  separate  clock  hour  estimates  of 
the  quantity  of  simulated  patient  encounters  was  offered;  the 
original  estimate  of  24  POI  hours  was  not  modified. 

While  the  reduction  in  scope  of  the  project  and  tne  prob¬ 
lem  of  whether  to  describe  it  in  terms  of  POI  hours  or  clock 
hours  are  interesting  observations,  it  is  more  important  to 
examine  the  proportion  of  the  total  materials  that  were  com¬ 
pleted.  In  July,  1Q75,  the  Sheppard  staff  reported  that 
while  Rll  of  the  first  trimester  lessons  had  been  finished, 
only  361  of  the  second  trimester  and  131  of  the  third  trimes¬ 
ter  lessons  were  in  a  usable  state.  The  remaining  lessons 
apparently  were  in  various  stages  of  completion  at  that  time. 

No  mention  is  made  by  Steinkerchner  et  al.  (1977)  as  to 
the  proportion  of  the  lessons  that  had  been  completed  at  the 
time  the  POMC  was  terminated.  Since  the  first  and  second 
trimester  materials  were  the  first  needed  for  student  use,  it 
may  be  assumed  that  the  development  effort  was  concentrated 
on  getting  those  lessons  produced.  If  they  were  completed 
(and  we  don't  know  that  they  were),  it  would  have  left  the 
bulk  of  the  third  trimester  lessons  to  be  developed  in  a 
relatively  short  period  of  time.  Had  the  POMC  phase  been 
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continued,  it  is  conceivable  but  not  likely  that  the  third 
trimester  lessons  could  have  been  completed  in  time.  Based 
on  our  observations  of  the  time  spent  by  the  Sheppard  staff 
in  completing  earlier  lessons,  the  task  of  finishing  the  53 
POI  hours  of  clinical  and  basic  science  materials  plus  the 
remaining  cases  in  the  series  of  simulated  patient  encounters 
would  have  required  lowered  standards  of  quality,  more  super¬ 
ficial  coverage  of  topics,  and  even  more  hours  of  overtime 
work  per  day  than  had  already  been  logged. 

The  extreme  time  pressure  to  complete  the  POMC  lessons 
posed  a  continuous  and  vexing  problem  to  the  project  staff. 
There  were  a  number  of  underlying  factors  to  this  plight, 
not  the  least  of  which  was  an  unrealistic  expectation  at  the 
start  of  the  project  as  to  the  time  required  to  produce  a 
given  body  of  materials.  In  the  original  request  for  train¬ 
ing  research  (see  Appendix  A),  the  project  initiator  assumed 
a  production  rate  of  3n0  hours  of  author  time  to  research, 
develop,  and  test  each  student  contact  hour  of  CBE  materials. 
(This  in  itself  is  not  necessarily  a  bad  initial  estimate.) 

He  then  went  on  to  compute  a  total  time  requirement  of  60,000 
author  hours  to  produce  200  hours  of  material  and,  assuming  a 
three  year  project  and  2000  work  hours  per  man  year,  deter¬ 
mined  that  a  manpower  requirement  of  ten  authors  was  estab¬ 
lished.  Again,  there  are  no  problems  with  the  fundamental 
arithmetic  of  these  calculations.  However,  this  analysis 
failed  to  take  account  of  other  non-authoring  duties  of  the 
staff  and,  more  importantly,  the  fact  that  the  proposed  pro¬ 
ject  milestones  only  allowed  18  months  for  development  prior 
to  the  start  of  the  evaluation  phase.  Clearly  a  staff  of  10 
should  not  have  been  expected  to  produce  200  hours  of  new 
materials  in  an  18  month  period.  The  job  was  too  big  for 
the  number  of  people  assigned  to  the  project  even  if  there 
had  been  no  hitches. 
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Unfortunately,  the  effort  was  plagued  by  several  time- 
consuming  problems  in  its  initial  stages.  First,  the  project 
initiator  left  the  project  just  at  the  time  it  was  officially 
getting  underway.  Since  he  was  the  originator  of  the  POMC 
concent  at  Sheppard,  his  departure  left  the  PLATO  staff  to 
struggle  on  its  own  in  interpreting  and  redefining  the  pro¬ 
ject's  goals . 

The  definition  process  was  complicated  by  the  staggered 
arrival  of  key  staff  members.  Although  the  project  approval 
date  was  November,  1973,  the  physician  chosen  to  serve  as 
subject  matter  specialist  and  curriculum  planner  did  not  join 
the  staff  until  July,  1979.  Many  important  decisions  about 
the  curriculum  were  postponed  until  his  arrival,  and  because 
the  choices  he  made  amounted  to  a  change  in  the  project's  goals, 
the  planning  and  lessons  developed  to  that  point  were  not 
utilized.  Even  after  the  physician's  arrival,  there  was  a 
shortage  of  manpower  for  lesson  development.  Four  of  the  ten 
staff  members  identified  had  yet  to  join  the  project,  with 
the  last  not  arriving  until  November,  1979. 

Perhaps  the  next  most  tangible  factor  underlying  the 
failure  to  complete  the  POMC  was  the  decision  to  use  only 
one  fourth  of  the  PA  class  in  the  experimental  group.  Realiz¬ 
ing  that  an  n  of  16  was  dangerously  small,  the  staff  decided 
to  replicate  the  experiment  with  a  second  class  of  PA  students 
to  start  the  sequence  the  following  trimester.  But  in  order 
for  two  classes  to  have  time  to  complete  the  entire  POMC 
within  the  scheduled  evaluation  period,  it  was  necessary  to 
move  the  start  of  the  implementation  forward  to  July,  1975, 
reducing  the  time  available  for  lesson  development  by  six 
months.  The  scope  of  the  task  had  been  reduced  somewhat  by 
this  time  with  plans  to  develop  195  POI  hours  of  CBE  materials 
rather  than  the  200  contact  hours  originally  proposed.  Never¬ 
theless,  the  task  was  still  too  large  to  be  completed  in  view 
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of  the  late  arrival  of  the  staff  and  the  severely  compressed 
time  schedule  forced  upon  them  by  the  scheduling  constraints 
of  the  PA  students. 

There  were  other  factors  which  had  subtle  but  real 
effects  on  the  project's  rate  of  lesson  production.  One  of 
the  more  important  of  these  was  the  fact  that  many  of  the 
authors  were  considered  to  be  basic  scientists  having  a  well- 
defined  subject  matter  specialty  ( e  .  g  .  ,  chemistry,  anatomy, 
pharmacy,  clinical  psychology).  Although  each  was  a  subject 
matter  expert  in  his  own  right,  most  lacked  the  broader  per¬ 
spective  afforded  by  clinical  experience.  Placed  as  they 
were  in  a  situation  of  trying  to  present  material  in  a  clini¬ 
cal  context  (i.e.,  illustrating  its  clinical  relevance),  these 
basic  scientists  were  at  a  real  disadvantage.  They  had  to 
depend  on  the  physician  and  another  staff  member  who  was 
a  recent  graduate  of  the  PA  course  for  assistance  in  this 
regard.  Because  the  basic  scientists  had  limited  experience 
in  med ic al/cl inic al  matters  relating  to  the  respiratory  sys¬ 
tem,  they  had  to  spend  a  great  deal  of  time  in  researching 
and  organizing  the  topics  they  were  teaching,  and  hence  they 
were  unable  to  develop  materials  as  rapidly  as  they  otherwise 
might  have.  This  problem  came  about  because  the  objectives 
of  the  project  were  changed  after  its  inception,  although 
staff  was  selected  and  hired  on  the  basis  of  the  original 
goals. 

Other  problems  hindering  the  lesson  development  process 
included  a  general  lack  of  experience/training  in  instructional 
design  procedures,  the  fact  that  only  two  staff  members  had 
taught  portions  of  the  PA  course  before,  only  three  persons 
had  previous  teaching  experience,  and  none  had  had  previous 
experience  in  designing  a  curriculum.  It  might  also  be  said 
that  talents  of  some  staff  members  were  not  fully  utilized  in 
lesson  production.  The  Instructional  Systems  Development 
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specialist,  the  illustrator,  and  one  of  the  experienced  com¬ 
puter  programmers  probably  could  have  been  utilized  more 
fully  in  an  overall  team  effort  for  production. 

In  summary,  the  task  of  developing  and  implementing 
such  large  amounts  of  new  curriculum  materials  was  highly  over- 
ambitious  and  unrealistic  given  the  manpower  limitations  and 
time  constraints  imposed  on  the  project.  Progress  was  further 
limited  by  the  staggered  arrival  of  staff,  the  change  in 
project  goals  implied  by  a  reinterpretation  of  the  DOMC  concept, 
the  fact  that  the  basic  scientists'  subject  matter  expertise 
was  nullified  by  the  decision  to  focus  narrowly  on  one  organ 
system,  and  failure  to  utilize  fully  the  talents  of  the  avail¬ 
able  staff.  It  was  only  through  much  hard  work  and  many  over¬ 
time  hours  that  the  staff  was  able  to  complete  as  much  course¬ 
ware  as  they  did. 

Di f f icu Ity  of  Defin ing  the  POMC 

The  third  major  outcome  of  the  Sheppard  project's  initial 
phase  was  that  not  much  was  learned  (or  at  least  not  much  was 
documented)  about  the  nature  of  the  POMC  concept  and  its 
feasibility.  It  is  not  clear  that  the  curriculum  that  was 
implemented  was  truly  problem  oriented. 

The  person  most  knowledgable  about  the  original  concept 
of  a  problem-oriented  reorganization  of  the  Physician  Assis¬ 
tant  curriculum,  the  project  initiator,  left  the  military 
just  at  the  time  the  Sheppard  CBE  project  was  given  official 
approval.  The  primary  document  which  we  have  available  des¬ 
cribing  his  conception  of  the  POMC  is  the  Request  for  Training 
Research  (see  Appendix  A).  Unfortunately,  because  of  its  brev¬ 
ity  that  document  does  little  to  edify  the  reader  regarding 
the  original  view  of  what  would  constitute  a  POMC.  A  later 
document  (22  June  1973)  outlined  a  reorganization  of  the  entire 
first  year  of  the  PA  program  under  a  group  of  24  sets  of 
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patient  complaints.  Under  this  scheme,  the  problem  presented 
by  the  patient  would  serve  as  a  touchstone  for  instruction  in 
basic  science  and  clinical  material  relevant  to  that  partic¬ 
ular  set  of  symptoms. 

By  the  time  of  the  February,  197*1,  planning  session  at 
Sheppard  AFB,  the  PLATO  staff  and  other  SHCS  representati ves 
had  decided  not  to  convert  the  entire  PA  curriculum  to  a 
problem-oriented  (symptom-oriented)  format,  citing  the  sheer 
size  of  the  task  as  being  beyond  the  scope  of  the  pro/ject. 
Instead,  they  adopted  the  following  working  definition  of 
the  POMC  (Kimball,  Note  1). 

A  problem-or iented  curriculum  is  the  grouping  of  course 
content  under  clinical  problems  and  the  presentation  of 
material  in  a  clinically-oriented  mode  such  that  the 
clinical  relevance  of  the  course  content  is  stressed  in 
order  to  develop  problem  solving  skills,  patient  manage¬ 
ment  and  investigative  skills,  and  the  collection  of  a 
medical  data  base. 

Included  in  this  curriculum  are  certain  basic  science 
materials  that  cannot  be  directly  related  to  clinical 
problems.  This  material  will  be  included  in  the  curricu¬ 
lum  as  prerequisites  to  bring  students  up  to  a  minimal 
acceptable  level  of  knowledge.  To  the  maximum  extent 
possible  this  material  will  be  presented  in  such  a  way 
as  to  contribute  to  the  development  of  problem  solving 
skills  through  the  use  of  inquiry  learning  techniques. 

A  passage  from  a  later  document  (dated  August,  197*4) 
reflects  the  staff's  decision  to  focus  on  topics  relating  to 
the  respiratory  system.  While  it  is  consistent  with  the  work¬ 
ing  definition  of  a  POMC  quoted  above,  it  provides  some  addi¬ 
tional  insight  as  to  the  project's  evolving  plans. 


Patient  problems  (i.e.  the  symptoms  or  complaints  for 
which  medical  attention  is  sought)  will  be  defined  and 
used  as  a  point  of  departure  to  teach  the  basic  and 
clinical  science  material  related  to  the  Respiratory 
System . 

This  will  lead  to  student-computer  interaction  to  solve 
a  variety  of  clinical  problems  (i.e.  exercises  in  differ¬ 
ential  diagnosis  and  therapeut •  s) . 


Thus,  the  curriculum  will  be  composed  of  a  number  of 
integrated,  but  potentially  free-standing  lessons  deal¬ 
ing  with  the  basic  and  cTinic'al  scYencesas  well  as  sim¬ 
ulated  patient  encounters.  The  student  will  have  access 
to  basic  and  clinical  science  review  material  from 
anywhere  within  the  lesson  sequence. 

The  last  portion  of  this  statement  emphasizes  the  staff's 
expectation  that  students  would  be  able  to  branch  back  to 
study  or  review  basic  science  material  relevant  to  specific 
cases . 

A  September,  1974,  document  gave  the  following  elabora¬ 
tion  of  the  group's  interpretation  of  "problem  or ientation" : 

The  term  "problem-or iented  curriculum"  has  been  taken  to 
mean  several  things:  (1)  a  clinically  relevant  curriculum 
heavily  based  on  case  descriptions,  using  those  both  as 
a  point  of  departure  and  as  a  point  of  reference  for 
teaching  the  associated  medical  material.  This  naturally 
implies  (2)  an  integrated  approach  including  the  areas 
of  anatomy,  microanatomy,  biochemistry,  genetics,  immu¬ 
nology,  neuroscience,  physiology,  pathology/pathophysiol¬ 
ogy  and  clinical  medicine,  pharmacology/therapeutics , 
and  laboratory  techniques.  In  addition,  there  will  be 
practice  in  solving  clinical  problems  (making  diagnoses 
and  treatment  plans  for  simulated  cases) . 

Yet  another  descriptive  statement  is  given  in  the  POMC 
Implementation  Plan.  Although  the  entire  plan  is  included 
as  Appendix  R,  the  relevant  section  is  restated  here  to  aid 
comparison : 

A  problem-oriented  medical  curriculum  is  defined  as  the 
grouping  of  course  material  under  clinical  topics  where 
the  clinical  relevance  of  course  content  can  be  stressed. 
Emphasis  is  placed  on  developing  problem  solving  skills 
in  diagnosis  and  treatment  of  disease.  To  facilitate 
development,  the  POMC  has  been  divided  into  two  parts. 
Part  I  presents  the  relevant  basic  science  and  clinical 
material  in  an  integrated  manner  supported  by  illustra¬ 
tive  case  material  and  using  clinical  subjects  as  point 
of  departure  to  teach  basic  sciences.  Part  II  will 
consist  of  "Simulated  Patient  Encounters"  which  will  be 
approached  as  unknowns  by  the  students.  Diagnostic  and 
treatment  skills  as  well  as  clinical  problem  solving 
skills  will  be  amplified  in  Part  II. 
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All  of  these  statements  are  fairly  similar,  but  the  subtle 
differences  may  be  particularly  important  in  view  of  later 
author  confusion  and  disagreement  over  what  a  prob lem-or iented 
curriculum  should  entail.  As  one  author  said,  "There  are  as 
many  definitions  of  'problem  oriented'  as  there  are  authors." 

It  is  readily  apparent  that  the  staff  did  not  prepare  a 
carefully-stated  operational  definition  of  the  POMC  which 
would  serve  as  a  touchstone  for  ail  development  efforts. 

The  importance  of  clinical  relevance  is  stressed  in  all 
of  these  statements,  but  thev  differ  as  to  what  place  clinical 
cases  should  have  in  the  curriculum.  This  was  a  substantial 
point  of  author  disagreement.  Authors  did  not  agree  on  the 
question  of  whether  individual  lessons  must  necessarily  include 
cases  (or  clinical  references),  or  even  problem-solving 
e 1 ements . 

One  author  thought  that  a  curriculum  which  included 
patient  simulations  could  thus  be  considered  "problem  oriented" 
regardless  of  whether  other  lessons  in  the  curriculum  centered 
on  clinical  cases  or  conditions.  According  to  this  criterion, 
Sheppard's  curriculum  was  indeed  "problem  oriented",  as  would 
be  any  others  which  include  case  simulations! 

Other  authors  thought  the  curriculum  should  proceed 
backward  from  clinical  cases  to  the  basic  science  material 
necessary  to  diagnose  them,  finally  ending  in  having  the 
student  diagnose  simulated  cases. 

In  a  thorough  review  of  those  lessons  known  to  have  been 
studied  by  PA  students,  we  determined  that  only  about  seven 
percent  included  clinical  cases  or  simulations.  The  project 
staff  outlined  the  following  criterion  for  a  "problem-oriented 
curriculum  in  a  September,  1974,  document:  "  ...  a  clinically 
relevant  curriculum  heavily  based  on  case  descriptions  using 
those  both  as  a  point  of  departure  and  as  as  a  point  of  ref¬ 
erence  for  teaching  the  associated  medical  material  ...". 
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According  to  this  standard,  the  Sheppard  curriculum  would 
probably  not  be  considered  problem  oriented  because  such  a 
small  fraction  of  lessons  fit  this  description. 

Several  factors  contribute  to  this  problem.  First, 
changes  in  project  administration  and  the  delayed  arrival  of 
the  senior  content  expert  (who  exercised  authority  over  cur¬ 
riculum  definition  and  development),  resulted  in  considerable 
rehashing  and  revamping  of  both  the  definition  of  "problem 
orientation",  and  the  implementation  plan  itself.  For  example, 
in  the  summer  of  1974,  when  the  final  group  of  Sheppard  staff 
visited  CERL  for  training  in  the  TUTOR  language,  they  reported 
that  the  definition  of  "problem  orientation"  was  still  being 
determined.  This  was  nearly  a  year  after  the  inception  of 
the  Sheppard  project. 

Second,  due  to  the  lack  of  firm  administrative  policy  or 
control ,  authors  exercised  their  own  judgment  over  what  "prob¬ 
lem  oriented"  meant,  and  how  that  related  to  their  own  lessons. 
Consequently,  there  is  little  curriculum  uniformity  either  in 
terms  of  "problem  orientation"  or  in  the  approach  to  (or 
necessity  for)  teaching  problem-solving  skills. 

In  a  memorandum  summarizing  the  discussions  and  decisions 
made  during  the  February,  1974,  planning  session,  the  acting 
project  director  said  that  the  "  ...  primary  objective  of  the 
project  is  to  reach  some  conclusions  regarding  the  instruc¬ 
tional  effectiveness  of  the  'concept'  of  a  problem  oriented 
curriculum"  (Kimball,  Note  1).  Judging  from  the  variety  of 
perceptions  of  what  constituted  a  POMC,  the  "concept"  of  a 
problem-oriented  curriculum  was  never  clearly  defined.  It  is 
our  belief  that,  in  the  absence  of  unambiguous  criteria  for 
determining  what  would  and  would  not  qualify  as  a  POMC,  the 
question  of  the  instructional  effectiveness  of  the  POMC 
"concept"  could  not  have  been  determined  even  if  all  lesson 
materials  had  been  completed  and  properly  implemented. 
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Limitations  of  jthe  Fvaluation  Plan 

A  fourth  major  area  for  analysis  among  the  outcomes 
the  PO^C  phase  is  that  of  the  planning  and  execution  of  the 
evaluation.  Since  a  major  goal  of  the  project  was  to  evalu¬ 
ate  the  concept  of  a  POMC,  it  is  important  to  examine  the 
plans  and  procedures  that  were  to  be  applied. 

Several  points  regarding  the  evaluation  plans  and  staff¬ 
ing  are  noteworthy.  First,  there  was  a  shortage  of  evaluation 
expertise  on  the  project  staff.  The  person  who  was  designated 
as  the  evaluator  was  a  clinical  psychologist  who  had  come  to 
the  project  with  the  expectation  that  he  would  work  as  an 
author  on  those  curriculum  topics  having  to  do  with  the 
psychology  of  patient  care.  When  the  project's  goals  were 
defined  more  narrowly  in  terms  of  respiratory  topics,  this 
individual  was  assigned  the  responsibility  of  designing  and 
implementing  the  evaluation.  He  had  verv  little  formal  train¬ 
ing  in  data-based  evaluation  procedures,  but  was,  of  course, 
well  trained  in  observation  and  interview  techniques.  These 
skills  proved  quite  useful  in  diagnosis  of  student  complaints 
and  in  determining  their  reasons  for  asking  to  be  dropped 
from  the  POMC  condition. 

Perhaps  because  of  the  relative  lack  of  local  evaluation 
expertise,  the  Sheppard  project  staff  had  the  continuing  expec¬ 
tation  that  representatives  of  other  agencies  would  supplement 
their  own  evaluation  efforts.  In  particular,  it  was  believed 
from  the  outset  that  AF.HRL  would  take  an  active  role  in  struc¬ 
turing  and  carrying  out  the  evaluation.  According  to  the 
Research  Agreement,  AFHRL  project  monitors  were  to  assist  in 
the  measurement  of  student  and  instructor  attitudes  as  well 
as  student  performance.  This  expectation  was  reinforced  at 
an  August,  1 9  7 ,  evaluation  conference  hosted  by  AFHRL.  Fur¬ 
thermore,  it  was  learned  at  this  conference  that  ARPA  had 
contracted  with  the  Educational  Testing  Service  (FTS)  to  con- 


31 


1 


duct  an  overall  evaluation  of  the  use  of  the  PLATO  IV  system 
at  ARPA-suppor ted  military  training  sites  (this  responsibility 
was  later  given  to  CFRL's  MTC  and  PEER  groups.  As  part  of 
its  evaluation  role,  ETS  was  to  prepare  a  list  of  clinical 
problems  and  to  assist  in  the  measurement  of  clinical 
problem-solving  skills.  These  hopes  were  later  made  more 
specific,  as  can  be  seen  in  the  POMC  evaluation  plan  (Appen¬ 
dix  C).  According  to  this  plan,  ETS  was  to  provide  items  for 
an  examination  covering  respiratory  disease.  If  insufficient 
items  were  available  from  ETS  item  banks,  yet  another  outside 
agency  (the  University  of  Nebraska  Physician  Assistant  program 
staff)  would  be  called  upon  to  supply  items.  ETS  was  also 
expected  to  provide  some  "simulated  patient  encounters"  in 
paper-and-penc il  form  to  be  used  to  measure  clinical  problem¬ 
solving  skills  of  both  the  POMC  and  Control  grouo  students. 

It  is  clear  from  Appendix  C  that  the  Sheppard  staff  was 
banking  heavily  on  support  from  outside  agencies  and  that 
they  were  willing  to  assume  that  these  agencies  would  be  able 
to  carry  through  on  their  assigned  responsibilities.  It 
should  be  noted,  however,  that  at  the  time  of  the  preparation 
of  the  evaluation  plan  (March,  1975)  very  little  work  had 
been  comoleted  toward  the  assembly  and  development  of  the 
measurement  instruments.  This  was  in  spite  of  the  fact  that 
the  evaluation  period  was  to  begin  in  July,  1975. 

The  evaluation  plan  itself  called  for  nine  studies. 

These  were:  (a)  a  comparison  of  POMC  and  Control  groups'  aca¬ 
demic  knowledge  (of  respiratory  material),  (b)  a  comparison 
of  their  problem-solving  skills,  (c)  examination  of  the  POMC 
group's  attitudes  toward  computer-based  education,  (d)  moni¬ 
toring  of  the  POMC  group's  performance  on  lesson  and  block  exams, 
(e)-(g)  studies  of  authors'  performance,  attitudes/opinions, 
and  characteristics,  (h)  summary  of  costs,  and  (i)  a  study 
of  the  reliability  of  the  PLATO  IV  system  and  terminals. 
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Mad  they  all  been  completed,  these  nine  studies  would 
have  addressed  worthy  evaluation  goals.  However ,  it  is  doubt¬ 
ful  whether  many  of  them  could  have  provided  much  general- 
izable  information.  There  are  several  reasons  for  this 
assessment . 

In  the  case  of  the  comparative  study  of  academic  know¬ 
ledge,  for  example,  the  sole  basis  for  comparison  was  to  be 
a  comprehensive  test  over  basic  science  and  clinical  concepts 
relating  to  the  respiratory  system  and  its  diseases.  The 
plan  called  for  some  empirical  development  work  in  the  con¬ 
struction  of  the  test,  but  it  is  not  clear  that  sufficient 
work  could  have  been  done  in  the  allotted  time  and  with  the 
available  subject  pool  to  assure  adequate  reliability  and 
validity  for  the  desired  purposes.  When  such  heavy  reliance 
is  placed  on  a  single  instrument,  it  is  essential  that  suffi¬ 
cient  development  effort  be  invested. 

Another  set  of  concerns  underlying  the  academic  compari¬ 
son  study  is  that  it  was  to  be  conducted  under  highly  reactive 
arrangements.  Because  the  implementation  constraints  allowed 
for  such  a  small  n,  the  POMC  group  was  tres-ed  much  differ¬ 
ently  than  the  Control  group  in  regard  to  scheduling  and  test¬ 
ing.  These  reactive  arrangements  created  additional  stresses 
for  the  POMC  group  which  may  have  interfered  with  (or  enhanced) 
that  group's  learning  of  the  target  material:  hence,  the 
reactive  arrangements  represent  confounding  factors  which 
would  have  made  interpretation  of  comparative  results  quite 
difficult.  Other  potentially  contaminating  influences  can  be 
traced  to  the  small-group  study  patterns  followed  by  the  PA 
students.  It  is  likely  that  POMC  and  Control  group  students 
studied  together  in  the  evening  hours  and  on  weekends.  If 
so,  it  is  quite  possible  that  there  was  a  sharing  of  notes, 
recollections,  etc.  which  may  have  acted  to  "wash  out"  any 
differential  treatment  effects.  Although  it  is  reasonably 
certain  that  Control  group  students  did  not  actually  study 
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pOMC  lessons  on  the  PLATO  system,  they  may  have  had  some 
vicarious  experience  with  them  through  their  friends  in  the 
POMC  group.  It  is  even  more  likely  that  the  POMC  students 
may  have  studied  lecture  notes  for  those  classes  they  had 
"missed"  while  attending  PO^C  sessions  in  the  PLATO  classroom. 
At  any  rate,  without  adequate  controls  to  assure  the  "purity" 
of  the  treatment  conditions,  it  would  be  difficult  to  inter¬ 
pret  the  results  of  the  comparison  or  to  generalize  them  to 
other  settings.  Furthermore,  because  the  POMC  was  adminis¬ 
tered  entirely  on  the  PLATO  system  and  the  Control  curriculum 
was  presented  entirely  by  traditional  lecture  approaches, 
there  was  a  complete  confounding,  between  strategy  (i.e., 
problem-oriented  versus  traditional)  and  mode  of  presentation. 
Had  one  group  outperformed  another  on  the  academic  test,  it 
would  be  unclear  whether  it  had  been  because  of  differences 
in  curricular  approach  or  differences  in  presentation  medium. 
This  PLATO-POMC  confounding  was  a  pervasive  problem  and  one 
which  was  not  addressed  by  Sheppard's  evaluation  plan. 

Of  the  remaining  eight  studies  that  were  planned,  only 
those  dealing  with  authors'  performance,  attitudes/opinions, 
and  characteristics  are  singled  out  for  comment  here.  As  a 
whole  these  studies  represent  legitimate  areas  of  inquiry, 
but  because  of  the  complexity  of  such  notions  as  "author  per¬ 
formance"  and  "author  attitudes",  they  are  very  difficult  to 
pursue  systematically.  For  example,  author  performance  was 
to  be  measured  by  judgments  of  lesson  quality,  supervisory 
ratings,  and  peer  ratings.  The  dimensions  of  TUTOR  language 
progr amming  ability,  lesson  design  efficiency,  and  creativity 
were  to  be  the  primary  indices  of  "performance",  and  it  was 
to  be  the  task  of  AFI1RL  researchers  to  develop  suitable  scales 
and  measures.  To  our  knowledge,  this  particular  study  was 
not  seriously  attempted--perhaps  because  of  the  great  diffi¬ 
culty  in  defining  and  measuring  these  dimensions.  Likewise, 
we  know  of  no  significant  effort  to  measure  author  attitudes. 
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A  number  of  aptitude  test  scores  and  other  data  were  assembled 
on  the  author  group  to  identify  certain  key  characteristics, 
as  it  was  hoped  that  these  could  be  used  to  predict  what 
kinds  of  individuals  should  be  selected  as  authors  for  future 
projects.  However,  because  these  individuals  had  already 
been  selected  and  were  functioning  on  the  job,  it  does  not 
seem  likely  that  they  would  vary  widely  in  easily-measured 
dimensions.  Even  if  some  relationships  had  been  found  between 
author  characteristics  and  performance  in  this  selected  group, 
they  may  not  be  valid  for  choosing  from  the  general  population 
of  candidates.  The  goal  of  this  study  was  a  worthy  one,  but 
it  is  not  clear  that  much  of  predictive  value  could  have  been 
learned  from  such  a  small  sample  of  authors.  Perhaps  this  is 
why  the  results  of  the  study  were  not  reported  by  Ste in kerchner 
et  al.  (1977). 

In  addition  to  these  criticisms  of  what  was  planned  for 
the  evaluation,  we  can  also  point  to  some  areas  of  omission. 

It  is  clear  that  the  staff  believed  their  primary  goal  was 
to  evaluate  the  effectiveness  of  the  POMC;  they  did  very 
little  to  evaluate  the  effectiveness  of  its  impl ementat ion . 
Because  of  the  many  problems  in  defining  and  developing  the 
new  curriculum  with  a  new  medium  under  tight  time  constraints 
and  the  rigidities  of  the  PA  course,  the  project  was  really 
one  of  attempting  to  initiate  a  number  of  innovations  in  a 
highly  traditional  and  structured  environment.  One  of  the 
prime  targets  of  opportunity  in  such  a  settine  is  the  docu¬ 
mentation  of  procedures  used  in  installing  the  innovations. 

In  the  POMC  evaluation  plan,  however,  there  were  no  provisions 
for  describing  the  communications  between  the  project  staff 
and  members  of  the  PA  staff.  Furthermore,  Steinkerchner  et 
al.  (1977)  did  not  report  on  those  important  transactions. 

The  likelihood  of  failing  to  examine  unanticipated  outcomes 
is  one  of  the  major  drawbacks  to  a  preordinate  approach  to 
evaluation  . 
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A  second  area  of  opportunity  that  was  not  sufficiently 
tapped  in  the  POMC  evaluation  was  that  of  determining  those 
military  training  applications  for  which  the  PLATO  system 
would  be  particularly  well  suited.  The  primary  goal  of  the 
project  was  to  implement  and  test  a  prototype  problem-oriented 
curriculum.  The  PLATO  system  was  treated  as  a  given--a  medium 
that  had  already  been  Droven--rather  than  as  an  important  com¬ 
ponent  of  the  overall  innovation.  It  is  true  that  the  plans 
provided  for  an  assessment  of  costs  and  reliability  of  the 
PLATO  system,  but  they  did  not  provide  for  a  determination  of 
its  strengths  and  weaknesses  for  medical  education.  One 
result  o^  the  staff's  seeming  acceptance  of  PLATO  as  an  estab¬ 
lished  medium  is  that  they  implemented  all  POMC  instruction 
on  the  PLATO  system.  For  example,  they  did  not  differentiate 
between  instructional  units  that  required  large  amounts  of 
interaction  and  those  that  did  not--all  were  created  in  the 
form  of  CPF  lessons.  In  contrast,  it  should  be  noted  that 
the  plans  for  Phase  II  (see  Appendix  D)  called  for  a  much 
closer  examination  of  the  medium  and  its  suitability  for  use 
with  various  types  of  students  and  instructional  tasks. 

In  summary,  the  evaluation  of  the  POMC  phase  reflected 
the  constraints  imposed  by  the  PA  course  and  the  implemen¬ 
tation  plan  adopted  to  accommodate  those  constraints.  The 
evaluation  plan  was  conceived  to  fit  the  situation  rather 
than  defining  the  situation  to  ease  the  evaluation  process 
and  increase  the  usefulness  of  its  findings.  The  staff  put 
together  a  plan  that  fit  the  circumstances,  but  their  job  was 
made  more  difficult  by  their  own  inexperience  and  their  result¬ 
ing  dependence  on  other  agencies.  Furthermore,  because  of 
the  severe  constraints  of  the  course  and  the  narrow  definition 
of  evaluation  targets  (e.g.,  effectiveness  of  the  POMC  with 
little  emphasis  on  the  transactions  involved  in  its  implemen¬ 
tation),  the  gener al i zab il ity  of  the  evaluation  results  would 
have  been  limited  even  if  the  POMC  phase  had  not  been 
terminated . 
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Add  it iona 1  Outcomes 

Relations  between  PLATO  staff  and  PA  staff.  As  indicated 
earlier,  the  communications  between  the  PLATO  staff  and  the 
PA  instructors  were  not  a  target  of  the  SHCP./AFHRL  evaluation. 
Recause  CERL  did  not  have  access  to  the  PA  instructors  to 
conduct  interviews,  MTC's  information  about  their  attitudes 
toward  and  experiences  with  the  POMC  phase  is  also  limited. 
However,  through  our  discussions  with  the  Sheppard  staff  we 
have  gained  the  general  impression  that  communication  with 
the  PA  instructors  tended  to  be  sporadic  and  superficial. 

Many  of  the  PA  instructors  apparently  perceived  the  POMC/ PL  A TO 
project  as  being  a  temporary  and  experimental  arrangement 
having  no  long-term  impact  on  the  operation  of  the  PA  program. 
Consequently  they  showed  little  interest  in  the  project  and 
maintained  a  neutral  or  mildly  negative  attitude  toward  it. 
Initially  there  was  to  be  a  representative  from  the  PA  staff 
present  at  the  PLATO  project  staff  meetings.  Likewise,  the 
PLATO  group  was  to  send  representatives  to  the  PA  staff's 
meetings  to  act  as  liaison.  However,  the  importance  of 
maintaining  good  communications  between  the  two  groups  must 
not  have  been  fully  appreciated,  because  the  frequency  of 
"joint  attendance"  at  me°tings  soon  dropped  nearly  to  zero. 

Not  only  did  the  formal  channels  of  communication  remain 
underutilized  or  ineffective,  a  number  of  informal  approaches 
also  had  limited  success.  A  test  grade  averager  and  on-line 
grade  book  was  prepared  by  one  of  the  PLATO  staff  to  help  the 
PA  instructors  with  the  mechanics  of  grading  (made  more  diffi¬ 
cult  by  the  separate  make-up  exams  for  the  POMC  group).  In 
spite  of  its  apparent  convenience,  it  went  unused  and  unap¬ 
preciated  by  the  instructors.  Individual  authors  would  often 
go  to  their  counterpart  on  the  PA  staff  and  enlist  his  or  her 
help  in  reviewing  a  lesson  corresponding  to  a  lecture.  Though 
the  PA  instructors  were  generally  cooperative,  they  tended  to 
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view  the  PLATO  materials  as  a  nice  adjunct  but  not  as  a 

replacement  for  the  lectures.  In  a  few  cases  there  were 

/ 

some  disagreements  about  the  depth  of  coverage  that  should  be 
given.  These  arose  out  of  the  ambiguity  of  the  course's  Plan 
of  Instruction  and  a  changing  conception  of  the  role  of  a  PA. 

In  general,  however,  the  instructors  did  not  play  a  large  role 
in  determining  the  content  and  strategy  of  the  lessons. 

It  is  not  known  how  much  the  PA  staff's  apathy  may  have 
affected  the  students'  attitudes  or  performance,  but  it  was 
probably  not  a  positive  influence.  A  more  systematic  cultiva¬ 
tion  of  relationships  with  the  PA  staff  might  have  helped 
generate  a  more  favorable  attitude  and  greater  cooperation. 

On  the  other  hand,  given  the  fundamental  conception  of  the 
project  as  an  experimental  implementation  of  a  new  and  unproven 
style  of  curriculum,  the  PA  staff  may  not  have  been  "won 
over"  to  an  enthusiastic  and  positive  outlook  no  matter  what 
techniques  were  used. 

Interactions  between  MTC  and  Sheppard  staffs.  The  MTC 
group  (through  its  specially-designated  subgroup  "Shp  East") 
was  given  a  contractual  role  by  ARPA  to  provide  support  to 
the  Sheppard  staff  and  to  participate  with  them  in  develop¬ 
ment  of  the  POMC.  The  sponsor  had  visualized  MTC ' s  role  as 
one  of  active  participation--particularly  in  those  areas  of 
the  project  where  the  MTC  staff's  experience  could  lead  to 
significant  time  savings  (such  as  in  developing  the  programs 
to  implement  the  simulated  patient  encounters).  A  graduate 
of  the  SCHS  Physician  Assistant  program  was  assigned  by  the 
Air  Force  to  work  directly  with  MTC  staff  members  as  a  subject 
matter  expert.  It  was  hoped  that  his  presence  at  CF.RL  would 
enable  the  MTC  staff  to  contribute  directly  to  project 
planning  and  development. 

Although  working  relationships  between  the  MTC  and 
Sheppard  staffs  were  congenial  and  productive,  there  was 
a  reluctance  on  the  part  of  the  Sheppard  group  to  give  MTC 


the  role  of  co-participants  outlined  in  the  contract.  This 
may  have  been  partly  due  to  concerns  about  MTC's  lack  of 
specific  subject  matter  expertise  (accentuated  by  the  early 
departure  of  the  PA  assigned  to  assist  at  CERL)  .  Another 
obstacle  was  the  geographic  separation  between  Sheppard  AFB 
in  Wichita  Falls,  Texas,  and  CERL  in  Urbana,  Illinois.  Thus, 
with  some  exceptions,  MTC's  role  on  the  project  turned  out  to 
be  one  of  providing  advice  and  support  rather  than  one  of 
direct  participation. 

If  it  was  difficult  for  the  Sheppard  staff  to  consider 
MTC  as  co-participants,  it  was  even  harder  for  them  to  accept 
MTC/PF.ER  inputs  and  participation  as  evaluators  on  the  project. 
The  primary  intent  of  the  sponsor  was  for  evaluation  staff  at 
CF.RL  to  work  to  enhance  the  effectiveness  of  Sheppard's 
efforts.  It  seemed,  however,  that  the  Sheppard  group  perceived 
CERL's  evaluation  role  at  that  stage  as  being  summative 
rather  than  formative.  They  were  uncomfortable  with  the 
notion  of  "being  evaluated"  by  a  group  that  had  previously 
been  in  a  support  role  as  "co-participants".  As  a  result,  a 
considerably  less  interactive  role  was  negotiated  with  ARP A 
for  MTC  and  PEER  than  had  been  originally  anticipated. 

In  general,  the  impact  of  CERL's  involvement  with  the 
Sheppard  project  was  significant,  but  probably  not  as  great 
as  it  might  have  been.  Perhaps  the  greatest  factor  limiting 
MTC's  influence  was  the  geographical  separation  between  CERL 
and  Sheppard  AFB.  Other  obstacles  to  collaboration  (e.g., 
differences  between  military  and  civilian  methodologies  and 
MTC's  lack  of  medical  subject  matter  expertise)  would  have 
been  less  troublesome  had  the  two  groups  been  in  closer 
proximity.  For  example,  the  Sheppard  staff  stated  that  MTC 
reviews  of  their  lessons  would  have  been  more  helpful  had  the 
reviewers  been  located  at  Sheppard  ( Cal 1-Himwich ,  1977a). 
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OVERALL  ASSESSMENT 
Summary  of  Major  Outcomes 

The  most  important  (or  most  visible)  categories  of  out¬ 
comes  of  the  POMC  phase  were  discussed  in  the  previous  sections 
Of  these,  the  most  tangible  was  the  move  by  many  of  the  POMC 
group  students  to  be  dropoed  from  the  experimental  condition. 
The  primary  basis  for  their  request  was  a  dissatisfaction 
with  the  implementation  of  the  POMC  vis-a-vis  the  regular 
portions  of  the  PA  curriculum.  Being  a  minority  of  only  one 
fourth  of  their  class,  the  POMC  students  felt  threatened  by 
an  implementation  which  required  them  to  "miss"  lectures 
attended  by  the  majority  of  their  peers.  The  make-shift  test¬ 
ing  and  examination  procedures  were  also  a  source  of  discon¬ 
tent.  Another  complaint  was  that  the  POMC  appeared  to  be 
fragmented  and  lacking  in  continuity.  This,  of  course,  was 
a  further  result  of  the  perceived  necessity  to  replace  discrete 
hours  of  the  regular  course  with  new  materials  from  the  POMC. 
Some  problems  having  an  impact  on  student  attitudes  grew  out 
of  the  time  pressures  on  the  staff  and  hence  were  only  indi¬ 
rectly  related  to  the  implementation.  These  included  an 
unfortunate  introduction  to  the  project  (in  which  privileges 
were  first  offered  and  then  withdrawn)  and  programming  errors 
in  relatively  untested  (and  unvalidated)  lessons.  It  is  dif¬ 
ficult  to  assess  the  impact  of  the  latter  problems,  but  we 
believe  them  to  be  secondary  in  importance  to  the  fundamental 
constraints  on  the  implementation  imposed  by  the  PA  course 
itself. 

The  second  major  category  of  outcomes  was  the  complex 
of  factors  which  impeded  lesson  development  progress.  We 
believe  that  the  POMC  lesson  materials  may  not  have  been 
ready  in  time  for  use  even  if  the  first  project  phase  had  not 
been  terminated  early.  As  detailed  above,  several  factors 
contributed  to  this  situation.  They  include  an  unrealistic 


assumption  about  the  amount  of  materials  that  could  be  pro¬ 
duced  in  a  Riven  time  period,  the  late  and  staggered  arrival 
of  staff  members,  a  severely  compressed  time  schedule  due  to 
the  desire  to  run  a  second  group  of  16  students,  and  a  change 
in  project  goals  which  made  it  necessary  for  some  staff  mem¬ 
bers  to  write  materials  outside  their  own  specialties,  hence 
decreasing  their  efficiency. 

Another  clear  outcome  of  the  first  project  phase  was  that 
the  concept  of  a  problem-oriented  curriculum  was  not  opera¬ 
tionally  defined.  After  the  project  initiator  had  left  the 
service,  it  was  up  to  the  remaining  staff  members  to  settle 
on  a  definition  of  the  POMC.  They  never  did.  Although  work 
on  a  definition  began  in  February,  1974,  and  continued  through¬ 
out  that  year,  there  was  still  enough  disagreement  at  the  end 
of  the  first  phase  that  one  staff  member  stated  that  "there 
are  as  many  definitions  of  'problem  oriented'  as  there  are 
authors."  In  view  of  the  unresolved  ambiguity  as  to  what  con¬ 
stitutes  a  probl em-or iented  curriculum,  it  is  not  clear  that 
Sheppard's  Phase  I  materials  qualified  as  a  POMC.  Had  that 
phase  run  to  completion,  it  would  have  been  difficult  to  judge 
whether  or  not  the  concept  of  a  problem-oriented  curriculum 
had  been  implemented  and  tested. 

Furthermore,  even  if  the  student  sample  had  not  dimin¬ 
ished  to  the  verge  of  extinction,  if  the  POMC  lessons  had  been 
completed,  and  if  they  had  been  truly  representative  of  the 
concept  of  a  problem-or iented  curriculum,  there  were  several 
deficiencies  in  the  planning  and  conduct  of  the  evaluation 
which  would  have  limited  the  general  izabil ity  of  the  results. 

Conclusion 

It  is  clear  that  the  POMC  phase  of  Sheppard's  PLATO  pro¬ 
ject  did  not  meet  its  objectives.  Unfortunately,  the  innova¬ 
tive  nature  of  the  POMC  concept  may  have  been  more  suited  to 
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evolutionary  development  than  to  the  systems  approach  that 
was  attempted.  It  is  now  apparent  that  too  little  was  under¬ 
stood  at  the  outset  about  the  POMC  concept  and  how  it  should 
he  implemented  to  guide  the  project's  development.  This  was 
not  clear  at  the  beginning,  however,  and  the  negative  outcomes 
could  not  have  been  foretold.  In  spite  of  its  early  termina¬ 
tion,  the  POMC  phase  was  a  worthy  effort--a  number  of  lessons 
were  learned  which  provided  a  firmer  basis  for  the  follow-on 
project . 
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REQUEST  FOR  TRAINING  RESEARCH 

1.  FROM:  School  of  Health  Care  Sciences,  USAF 

Charles  N.  Mullican,  Lt  Col,  USAF,  MC 
Sheppard  AFB,  Texas  76311 

Autovon:  736-2210  A  April  1973 

2.  TITLE:  Evaluation  of  PLATO  IV  System  in  Air  Force  Medical  Training 

3.  PROBLEM:  A  critical  shortage  of  physicians'  services  exists  in  the  USAF 

Medical  Service,  reflecting  a  similar  nationwide  deficit. 
Traditional  modes  of  health  care  delivery  have  been  maximally 
stressed  and  found  inadequate;  new  categories  of  health  man¬ 
power  have  been  developed.  One  such  new  health  professional 
is  the  physician  assistant  (PA),  a  person  who  because  of  spe¬ 
cial  training  and  experience  can  perform  many  duties  formerly 
carried  out  only  by  physicians. 

The  allied  health  education  of  the  Physician  Assistant  and 
the  medical  education  of  the  physician  are  remarkably  similar 
in  content  and  scope,  therefore  most  PA  programs  are  modeled 
after  traditional  medical  school  curricula.  The  instructional 
approach  is  characteristically  one  in  which  there  is  a  hori¬ 
zontal  sequencing  of  basic  medical  sciences  with  clinical 
practice.  The  usual  modes  of  teaching  and  learning  are  lecture/ 
demonstrat ion  in  group/ lock-s tep  pattern.  The  major  difficulty 
with  such  a  curriculum  is  that  the  core  of  medical  knowledge 
which  must  be  memorized  is  usually  taught  without  a  fundamental 
frame  of  reference.  Basic  medical  sciences  are  often  presented 
in  isolation  from  their  clinical  relevance,  both  intellectually 
and  temporally.  Thus,  when  the  clinical  sciences  are  ultimately 
presented,  time-consuming  and  repetitious  review  of  the  basic 
sciences  must  precede  the  teaching  of  a  new  body  of  knowledge. 

The  basic  frame  of  reference  missing  in  a  traditional  medical 
curriculum  is  the  core  of  clinical  behavior  which  characterizes 
the  problem-solving  process.  Its  absence  has  heretofore  made 
the  application  of  recent  advances  in  educational  technology 
(systems  analysis  and  design)  difficult  and  inconsistent. 

4.  OBJECTIVE:  Two  major  objectives  of  this  study  are  envisioned:  (a)  The 

research  and  development  of  a  new,  problem-oriented,  medical 
curriculum  at  the  physician  assistant  level  using  an  advanced, 
computer-based  educational  system  and  (b)  a  comparison  eval¬ 
uation  of  the  instructional  effectiveness  of  this  curriculum 
vis-a-vis  the  present  traditional  curriculum. 


46 


5.  BENEFITS:  a.  Advanced  Research  Projects  Agency:  To  make  available  the 

development  of  a  fundamentally  new  problem-oriented  cur¬ 
riculum  upon  which  all  allied  health  instruction  nation¬ 
wide  can  be  constructed.  This  prototype  would  have  the 
potential  for  the  complete  reduction  of  all  present  time- 
consuming,  repetitious  and  costly  instructional  practices. 

b.  Department  of  Defense:  To  provide  an  instructional  test 
bed  for  the  new  generation  of  military  hospitals  project. 

c.  USAF/SHCS:  To  make  available  a  comprehensive  computer- 
based  educational  system  for  use  in  training  and  educat¬ 
ing  allied  health  professionals  in  the  USAF  medical  service 
through  the  utilization  of  existing  PLATO  IV  programs  as 
well  as  connection  with  the  tie  into  other  computer-based 
networks . 

d.  University  of  Illinois:  To  provide  CERL's  PLATO  IV  project 
wth  all  the  necessary  requirements  for  the  research  and 
development  of  the  types  of  hardware  necessary  to  meet  the 
unique  requirements  of  the  medical  education  system  by 
providing  the  data  base,  Instructional  Systems  Design,  test 
students,  and  evaluation  protocol. 

6.  MAJOR  USERS:  a.  SHCS 

b.  Other  uniformed  services 

c.  Civilian  educational  institutions 

d.  NGMH 

7.  TIME 

PHASING:  a.  Project  development:  Approval  data  +  18  months. 

b.  Test  evaluation:  Approval  data  +  30  months. 

c.  Final  report:  Approval  data  +  36  months. 

8.  UTILIZATION 

CONCEPT:  Because  the  development  of  a  new  curricular  approach  is  an 

evolutionary  process,  it  is  difficult  to  foretell  the  state 
of  the  art  at  the  time  of  Final  Report.  Nevertheless,  it  is 
conceivable  that  the  man-machine  interface  will  result  in  a 
continued  reinforcement  of  the  problem-solving  process  as 
the  dominant  mode  of  clinical  behavior.  Instructional  modules 
will  be  used  to  teach  a  basic  core  of  medical  knowledge  as  it 
relates  to  clinical  problems.  This  core  of  knowledge  will  be 
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9.  BRIEF  OF 

THE  PROJECT: 
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available  for  other  allied  health  personnel  to  aid  in  under¬ 
standing  their  own  roles  in  the  delivery  of  health  care,  that 
is,  their  place  along  the  diagnosis-treatment  loop.  Concur¬ 
rently  operating  will  be  a  continuous  evaluation  of  the  re¬ 
lative  merits  of  the  two  curricula,  with  and  without  PLATO  IV 


a.  Approach 

(1)  Givens: 

(a)  A  basic  core  of  medical  knowledge  for  USAF  Physi¬ 
cian  Assistants  exists  as  outlined  in  the  Plan 

of  Instruction,  Course  3ALR91730,  dated  3  August 
1972  (Append ix  A) . 

(b)  A  basic  pattern  of  clinical  behavior  character¬ 
izes  problem-solving  which  is  straightforward, 
logical,  and  scientific." 

(c)  A  charting  system  exists  for  recording  and  docu¬ 
menting  the  use  of  medical  knowledge  in  the 
solving  of  clinical  problems. 

(2)  Assumptions: 

(a)  Advances  in  educational  technology  and  instruc¬ 
tional  systems  analysis  can  be  greatly  aided  by 
a  computer  system  in  which  [9a(1)(a)  and  (b)] 
(above)  are  integrated  and  correlated. 

(b)  Such  a  system  will  be  self-paced  and  learner- 
oriented.  Evaluative  systems  will  constantly 
assess  the  effectiveness  of  instruction  and  the 
level  of  student  achievement.  Quality  instruction 
and  performance  will  not  be  compromised  in  de¬ 
ference  to  resources  conservation. 

(3)  Methodology 

(a)  Approximately  200  student  contact  hours  of  new 
curricular  material  will  be  developed  on  the 
University  of  Illinois'  CERL's  PLATO  IV  system. 

(b)  An  Instruct ional  Systems  Design  IAW  AFM  50-2 
will  be  developed  and  the  rough  outline  of  a 
problem-oriented  curriculum  wi 1  I  be  programmed 
into  PLATO  IV. 
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(c)  During  the  development  of  this  project,  students 
will  be  exposed  to  both  curricular  formats  in 
which  PLATO  IV  may  or  may  not  be  employed.  The 
suitability  and  contribution  of  computer-assisted 
instruction  will  be  evaluated  as  it  stands  alone 
and  vis-a-vis  the  curricula. 

(d)  The  attached  management  guide  for  a  patient  entering 
the  emergency  room  confused  or  in  coma  (Appendix  B) 
illustrates  how  the  problem-oriented  approach  may 

be  used  to  organize  curricular  material  into  data 
gathering  (Subjective  information  and  Objective 
data),  data  synthesis  (Assessment)  and  problem  re¬ 
solution  (Mans).  At  virtually  every  point  along 
the  problem-solving  loop,  branching  into  the  body 
of  basic  and  clinical  medical  knowledge  can  be 
made;  for  example,  while  studying  the  data  base 
contained  in  Subjective  information,  the  student 
can  review  methods  of  interviewing,  pertinence  of 
questions  to  be  asked,  and  so  on.  When  reviewing 
the  information  to  be  gathered  under  Objective  data, 
the  student  can  correlate  anatomy,  physiology,  radio 
logy  and  techniques  of  physical  examination.  Under 
Assessment,  the  student  is  appropriately  exposed 
to  classic  descriptions  of  disease  processes  which 
can  be  integrated  with  presentations  in  pathophysio¬ 
logy  and  biochemistry.  When  learning  of  the  problem 
resolution  phase  (Plan)  the  student  is  exposed  to 
the  fundamentals  of  laboratory  medicine  (clinical 
chemistry,  microbiology,  serology,  and  so  on).  Addi 
tionally,  pharmacology  is  rationally  interpolated 
at  this  point  to  describe  the  basic  knowledge  under¬ 
lying  medical  therapeusis. 

(e)  It  is  apparent  that  such  an  approach  stresses  the 
importance  of  a  certain  pattern  of  clinical  be¬ 
havior  and  it  is  this  critical  area  that  will  be 
subjected  to  intensive  evaluation  as  to  the  role 
of  CAI  in  developing  and  maintaining  desirable 
attitudes  and  dispositions. 

b.  Support  required 
(I)  Personnel 

3 00  hours  author  time  (AH)  are  required  to  research, 
develop  and  test  1  student  contact  hour  (SH). 


200  SH  =  60,000  AH. 
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]  man-year  =  2000  hours. 

Therefore  200  SH  =  30  man-years. 

For  a  three-year  project,  a  need  of  10  persons  is 
establ ished. 

(2)  PLATO  IV  terminals:  20  (15  student,  5  author). 

(3)  Dedicated  telecom  lines:  5 

(k)  Advisor-consultahts  at  the  University  of  Illinois:  2 

(5)  Physical  plant  and  equipment 

(a)  20  carrels 

(b)  Adequate  air  conditioning  to  keep  units  suffi¬ 
ciently  cool  (20,000  BTU  per  terminal). 

(6)  Installation  and  maintenance  of  PLATO  iV  equipment  at 
SHCS . 

(7)  Training  of  ten  SHCS  personnel  at  the  University  of 
Illinois  in  TUTOR  language,  curricular  development 
and  Instructional  capabilities  of  PLATO  IV. 

10.  SUMMARY:  A  new  curricular  approach  to  the  teaching  and  learning  of 

problem-oriented  health  care,  systems  analyzed  and  developed 
in  conjunction  with  an  advanced  computer-based  educational 
system,  is  described.  The  impact  upon  allied  health  education 
at  the  SHCS  is  a  1 1 -encompass i ng  since  such  a  curriculum  offers 
an  understandable  frame  of  reference  for  health  care  delivery 
as  it  relates  to  the  responsibilities  of  each  training  depart¬ 
ment.  The  research  design  will  attempt  to  provide  significant 
data  comparing  the  effectiveness  not  only  of  the  differing 
curricular  approaches  but  also  of  computer  assisted  instruction 
in  health  education. 


Hurst,  J.  W.  and  Walker,  H.  K. :  The  Problem-Oriented  System,  MEDC0M  Press, 
1972,  287  pages 
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POMC  IMPLEMENTATION  PLAN 

I.  References:  Coarse  Chart  3ALR91730  10  Feb  1975 

POI  3ALR9 1730 

Research  Agreement  "Evaluation  of  PLATO  IV 
System  in  Air  Force  Medical  Training" 

II.  Plan  Objective 

The  purpose  of  this  plan  is  to  provide  for  the  orderly  integration 
of  the  Problem  Oriented  Medical  Curriculum  (POMC)  developed  by  the 
PLATO  IV  Development  Branch  into  the  present  Physician  Assistant 
curriculum.  This  integration  will  be  done  in  such  a  way  that: 

1.  Disruption  of  the  present  curriculum  is  minimized. 

(a)  Steps  will  be  taken  to  insure  that  prerequisite  material 
is  taught  prior  to  the  time  it  is  required  as  a  basis  for  other  material. 

(b)  The  logical  flow  of  content  in  discipline  oriented  blocks 
will  not  be  jeopardized. 

2.  The  integrity  of  the  POMC  is  maintained,  i.e.  presentation  of 
material  is  not  excessively  fragmented. 

3.  No  degradation  in  training  will  result. 

4.  Evaluation  of  the  POMC  is  facilitated. 

III.  Scope 

This  plan  pertains  only  to  the  Physician  Assistant  Course  (Phase  I), 
3ALR91730.  The  time  frame  for  implementation  of  this  plan  is  Jun  1975  - 
Oct  1976  to  correspond  with  the  evaluation  phase  of  the  PLATO  IV  project. 

IV.  Background 

A.  On  4  Apr  73,  the  School  of  Health  Care  Sciences  submitted  a 
proposal  for  personnel  research  to  Hq  ATC.  The  resultant  research  agree¬ 
ment  calls  for  (1)  the  research  and  development  of  approximately  200 
hours  of  a  new  problem  oriented  medical  curriculum  at  the  physician 


assistant  level,  using  an  advanced  computer  based  education  system, 
and  (2)  the  evaluation  of  the  instructional  effectiveness  of  this 


curriculum  vis-3-vis  the  traditional  curriculum.  The  project  covers 
a  three  year  period  beginning  in  November  1973.  Evaluation  is  to  be 
completed  by  1  July  1976,  with  the  final  report  to  be  rendered  in 
November  1976. 

B.  A  problem  oriented  medical  curriculum  is  defined  as  the 
grouping  of  course  material  under  clinical  topics  where  the  clinical 
relevance  of  course  content  can  be  stressed.  Emphasis  is  placed  on 
developing  problem  solving  skills  in  diagnosis  and  treatment  of  disease. 
To  facilitate  development,  the  POMC  has  been  divided  into  two  parts. 

Part  I  presents  the  relevant  basic  science  and  clinical  material  in  an 
integrated  manner  supported  by  illustrative  case  material  and  using 
clinical  subjects  as  point  of  departure  to  teach  basic  sciences. 

Part  II  will  consist  of  "Simulated  Patient  Encounters"  which  will  be 
approached  as  unknowns  by  the  students.  Diagnostic  and  treatment 
skills  as  well  as  clinical  problem  solving  skills  will  be  amplified 
in  Part  II. 

C.  The  task  of  selecting  the  content  of  the  POMC  from  the  massive 
amount  of  material  included  in  the  present  physician  assistant  course 
was  undertaken  as  follows.  Selection  was  based  on  the  criteria  that 
the  clinical  area  would  (1)  accommodate  an  integrated  approach  to 
include  both  the  basic  and  clinical  sciences,  (2)  contain  fairly  cir¬ 
cumscribed  and  clinically  relevant  subject  matter,  (3)  provide  sufficient 
subject  matter  to  meet  the  project  objectives,  and  (4)  make  the  best  use 
of  resources  within  the  PLATO  IV  Development  Branch.  Jointly  exhaustive 
lists  of  possible  etiological  and  body  systems  were  tested  against  the 
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selection  criteria.  The  respiratory  system  (respiratory  disease)  was 
selected  as  the  best  subject  area  for  the  purposes  of  this  project. 

D.  A  list  of  basic  sciences  and  specific  clinical  topics  has  been 
developed  and  arranged  in  a  curricular  structure  which  defines  the 
basic  sequence  and  interrelationships  of  the  subject  matter  areas. 

Initial  planning/programming  has  begun  in  eight  of  the  proposed  20 
lesson  development  areas. 

V.  Approach  to  Implementation 

A.  Definitions  and  Constraints 

1.  Terminals.  There  are  16  PLATO  IV  student  terminals.  Prime 
time  (when  the  system  is  most  stable)  is  from  0740  to  2200  hours, 

Monday,  Wednesday  and  Friday,  0600  to  2200  Tuesday  and  Thursday,  and 
0800  to  1200  hours  Saturday.  The  total  available  terminal  hours  is  a 
constraint,  since  there  are  64  PA  students  in  any  given  semester,  plus 
30  PCNP  students  who  attend  PA  clinical  medicine  lectures. 

2.  POI  Hours.  For  planning  purposes,  hours  are  considered 
to  be  POI  hours  planned  for  a  particular  subject.  POI  hours  may  not 
necessarily  equate  to  hours  actually  spent  at  the  terminal  by  a  given 
student.  PLATO  material  at  the  lesson  or  lesson  sequence  level  is 
self-paced  and  variations  in  individual  students  can  be  expected. 

3.  Classroom  (C/L) /Complementary  Technical  Training  (CTT)  Hours. 
Since  computer  assisted  instruction  is  designed  to  teach  for  mastery  of 
the  material,  a  PLATO  IV  lesson  designed  to  meet  a  particular  POI  ob¬ 
jective  will  replace  both  the  associated  classroom  hours  and  CTT  hours. 

In  confuting  hours  in  the  present  curriculum  to  be  replaced  by  PLATO 
material,  both  C/L  and  CTT  hours  are  considered. 
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B.  Hours  Required  for  POMG  Material. 

1.  The  present  PA  curriculum  has  been  examined  to  identify 
those  hours  in  which  the  subject  matter  is  essentially  the  same  as 
POMC  content  and  can  therefore  be  replaced  by  the  comparable  POMC 
material.  A  detailed  breakdown  of  hours  by  POI  objective  by  semester 
is  included  in  Attachment  1.  A  summary  is  provided  in  paragraph  3 
below. 

2.  As  anticipated,  some  material  included  in  the  POMC  does 
not  have  direct  comparability  with  hours  in  the  present  curriculum. 
This  material  is  limited  to  "simulated  patient  encounters"  which  is 


expected  to  require  24 

student 

contact 

hours.  These 

hours  are  cons 

to  be  an  essential  part 

of  the 

POMC  for 

review  and  reinforcement. 

3.  Summary  of 

Hours 

Replacement  Hours: 

C/L 

CTT 

Sub-Total 

Total 

1st  trimester 

16.5 

7.5 

24 

2nd  trimester 

20.0 

5.0 

25 

3rd  trimester 

52.0 

20.0 

72 

Sub-total 

88.5 

32.5 

121 

121 

Additional  Hours: 

Simulations 

24 

Sub- total 

24 

Grand  total: 

145 

4.  In  the  original  research  proposal  an  arbitrary  estimation 
of  200  course  hours  was  designated  for  PLATO  IV  programs.  The  reduc¬ 
tion  in  these  hours  to  145  represents  the  integration  of  an  effective 
circumscribed  curriculum  into  the  present  course  with  the  least 
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disruption  of  continuity.  The  expansion  of  the  PLATO  hours  to  reach 
a  closer  approximation  of  the  original  estimation  of  hours  would  only 
serve  to  dilute  the  effectiveness  of  the  PLATO  project,  complicate 
the  evaluation  procedures  and  increase  the  problems  of  integration 
into  the  present  course. 

C.  Plan  for  Implementation 

1.  A  sample  of  16  Physician  Assistant  students  will  be  selected 
from  the  freshman  class  starting  in  June  1975  to  serve  as  a  pilot  group. 
Sampling  techniques  developed  by  AFHRL  will  be  used  to  select  the  16 
students.  A  second  sample  of  16  students  may  be  selected  from  the  class 
starting  in  Oct  1975  if  that  is  considered  desirable  on  the  basis  of 
evaluation  plans  and  experience  with  the  initial  group. 

2.  Students  in  the  sample  will  complete  PLATO  materials  in 
lieu  of  the  corresponding  work  in  the  current  curriculum.  They  will 
attend  all  of  the  scheduled  lectures,  except  those  that  have  been 
replaced  by  PLATO  lessons.  In  addition  to  criteria  listed  in  IV-C 

above,  considerable  concessions  have  been  made  in  the  selection  of  material 
to  be  included  in  the  POMC  so  that  discrete  lecture  hours  of  the  current 
curriculum  are  identifiable.  At  the  time  such  lectures  are  given, 
students  would  leave  the  classroom  and  report  to  the  PLATO  lab  for 
computer  based  instruction.  Overall,  the  same  criterion  objectives  will 
be  met  by  all  students,  but  the  temporal  sequencing  of  material  in  the 
problem  oriented  medical  curriculum  will  not  necessarily  parallel  that 
of  the  current  curriculum. 

3.  The  additional  hours  required  for  the  simulated  patient 
encounters  included  in  the  POMC  will  be  supplemental  hours  to  be  done  at 
the  student's  conveniences.  This  will  provide  review  and  reinforcement 
of  material  covered  in  Phase  I. 
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4.  Since  students  in  the  sample  group  will  not  necessarily  be 
working  on  the  same  subject  matter  at  the  same  time  as  the  rest  of  the 
class,  some  modification  of  the  normal  end  of  block  evaluation  procedures 
will  be  required.  POM C  students  would  take  the  normal  end  of  block 
tests,  but  questions  covering  coursework  taught  in  the  POMC  would  not 
be  counted  in  their  test  score.  The  PLATO  IV  staff  will  assume  respon¬ 
sibility  for  determining  student  proficiency  in  the  POMC  subject  areas 
on  a  semester  basis  to  determine  whether  or  not  the  student  is  competent 
to  proceed,  needs  remediation,  etc.  The  PLATO  staff  will  also  assume 
responsibility  for  needed  remediation  in  those  areas. 

VI.  Tasks  Remaining  to  be  Done 

A.  A  detailed  schedule  must  be  developed  for  the  present  curriculum 
that  identifies  specific  subjects  for  specific  lecture  hours,  and  also 
identifies  hours  during  which  POMC  material  is  to  be  taken  by  the  sample 
group . 

B.  Waivers  from  course  documentation  requirements  and  testing 
standards  must  be  requested  from  Hq  ATC. 

C.  Any  questions  involving  accreditation  from  the  University  of 
Nebraska  for  the  sample  group  must  be  resolved. 

1  Atch 

Breakdown  of  Replacement  Hours 
by  POI  Objective 


1st  Sem  -  Replacements 
P01  3ALR9 1730  (Vol  1) 


~ - 

1 

4 

57 


3rd  Sera  -  Replacements 
POI  3ALR9 1730  (Vol  3) 
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Define  the  air  bronchogram.  (.5) 


3rd  Semester  (cont'd) 
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22a  Explain  the  actions,  effects  and  9/5  Chemotheraphy 

uses  of  drugs  used  to  combat 
infection. 


Present  POMC 

Block  Paragraph  Objective  Hours  Subject  Hours 
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chronic  obstructive  lung  disease 
and  describe  pathophysiology  of 
cystic  fibrosis. 
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APPENDIX  C 

POMC  Evaluation  Plan 
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PROBLEM  ORIENTED  MEDICAL  CURRICULUM 
EVALUATION  PLAN 

(prepared  by  AFHRL/TT  as  a  result  of  evaluation  conference  held  6-7 
Mar  75  at  Sheppard  AFB) 

BACKGROUND 

Under  the  auspices  of  the  Advanced  Research  Projects  Agency  (ARPA) 
and  Air  Training  Command  (ATC) ,  a  Problem-Oriented  Medical  Curriculum 
supported  by  PLATO  IV  technology  is  being  introduced  in  the  School  of 
Health  Care  Sciences.  PLATO  can  provide  individualized  instructional 
material  presented  through  a  computer  controlled  plasma  panel.  The 
essential  notion  behind  the  Problem  Oriented  Medical  Curriculum  (POMC) 
is  that  academic  subject  matter  should  not  be  introduced  in  isolated 
units,  but  rather  in  the  context  of  clinical  diagnosis  of  actual  patient 
problems.  In  this  fashion,  it  is  anticipated  that  problem  solving  skills 
will  be  developed  to  a  greater  degree,  that  retention  of  information  will 
be  enhanced,  and  that  student  motivation  will  be  increased.  Details 
covering  introduction  of  this  innovation  into  the  Physician's  Assistant 
Course  can  be  found  in  the  POMC  Implementation  Plan,  10  Feb  75. 

PROBLEM 

The  goal  of  this  evaluation  plan  is  to  develop  a  set  of  procedures 
for  assessing  the  effectiveness  of  the  proposed  POMC. 

RATIONALE 

The  initial  step  in  the  process  is  to  define  a  rationale  which 
underlies  and  supports  the  entire  evaluation.  Proper  evaluation  is 
obviously  a  complex  matter,  for  the  concept  of  "effectiveness"  is 
clearly  multidimensional.  However,  two  sets  of  questions  seem  to  be 
generated  almost  immediately.  The  first  group  is  primary  and  basically 
operational  in  nature;  these  questions  are  designed  to  create  a  rational 


policy  for  or  against  full  scale  implementation  of  the  innovation. 

Simply  stated,  this  set  of  questions  asks: 

.  can  it  be  done? 

.  is  it  instructionally  effective? 

.  what  does  it  cost  (to  institute,  to  operate,  to  maintain)? 

.  is  it  reliable? 

A  secondary,  or  research-oriented  group  of  questions  is  also  of 
interest.  These  are  directed  at  explaining  the  success  or  failure 
of  the  innovation  in  the  particular  environment  studied.  This  set 
consists  of  asking 

.  how  was  it  done? 

.  why  was  it  effective? 

.  how  could  it  be  improved  (in  terms  of  costs  lowered  or  ef¬ 
fectiveness  increased)? 

These  latter  questions  are  not  unimportant,  but  higher  priority  must 
be  given  to  the  first  set  since  they  were  judged  to  be  more  relevant  to 
the  purposes  of  the  sponsoring  agencies  (ARPA  and  ATC) .  To  some  extent 
the  secondary  questions  will  be  addressed  in  the  final  report,  especially 
with  regard  to  describing  the  process  involved  in  setting  up  the  POMC, 
but  the  major  emphasis  of  available  resources  is  being  placed  on  answering 
the  first  group  of  questions. 

Another  issue  which  was  faced  in  planning  the  evaluation  concerns 
the  extent  to  which  comparative  measurement  will  be  made,  as  distin¬ 
guished  from  following  purely  descriptive  procedures.  This  distinction 
is  analogous  to  the  difference  between  norms  referenced  and  criterion 
referenced  measurement.  In  the  comparative  approach,  procedure  A  is 
compared  with  procedure  B  on  various  criteria  of  effectiveness,  while 
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in  the  descriptive  approach,  effectiveness  is  measured  with  respect  to 
meeting  standards  or  specifications  in  some  absolute  sense.  Each  approach 
has  its  own  peculiar  advantages,  but,  where  feasible,  it  was  decided 
to  use  comparative  measures  as  much  as  possible  in  order  to  contrast  the 
POMC  approach  with  traditional  instruction. 

Another  conceptual  consideration  important  for  this  rationale  deals 
with  focusing  on  which  aspects  of  the  situation  should  be  studied.  Although 
not  exhaustive  of  the  possibilities  inherent  in  this  situation,  three  major 
aspects  were  identified  for  intensive  study — students,  authors,  and  PLATO 
IV  itself. 

A  final  concern  involves  deciding  which  specific  characteristics  and 
variables  need  to  be  measured.  Since  student  and  author  performance  are 
crucial  to  judging  effectiveness,  these  naturally  are  emphasized.  But  it 
was  also  viewed  as  important  to  measure  attitudes  and  to  sample  opinions 
from  the  participants.  In  addition,  concern  with  author  background 
characteristics  and  their  relation  to  performance  was  seen  as  both  re- 
searchable  and  important  for  future  implementation.  With  respect  to  the 
PLATO  device,  its  various  cost  factors  and  overall  reliability  must  be 
addressed. 

The  foregoing  considerations  gave  rise  to  Fig.  1  which  outlines  the 
major  studies  to  be  completed  as  part  of  this  evaluation.  Two  further 
constraints  which  influenced  the  choice  of  particular  studies  were  (a) 
the  necessity  to  work  within  the  guidelines  of  the  POMC  Implementation 
Plan,  and  (b)  the  limited  manpower  resources  of  both  the  PLATO  Devel¬ 
opment  Branch  and  AFHRL. 

SAMPLING  CONSIDERATIONS 

For  those  studies  involving  comparative  evaluation  of  students,  a 
POMC  test  group  (numbering  16)  will  be  selected  from  personnel  entering 
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June  1975.  These  students  will  be  presented  specific  problem 
oriented  material  on  PLATO  IV,  rather  than  receiving  the  "equivalent" 
instructional  material  in  the  traditional  classroom  setting.  These 
materials  will  be  distributed  throughout  the  entire  three  semesters  of 
classwork  at  the  School  of  Health  Care  Sciences  (SHCS).  The  remaining 
48  students  from  the  June  1975  class  will  serve  as  a  control  group  and 
will  not  be  exposed  to  the  problem  oriented  material  related  to 
Respiratory  Disease  on  PLATO  IV. 

The  POMC  test  group  will  be  randomly  selected  from  the  PA  students 
entering  in  June  1975.  If  the  sample  chosen  by  this  method  does  not 
differ  significantly  from  the  control  group  on  SAT  composite  scores,  the 
sample  will  be  accepted  as  representative.  If  there  is  an  appreciable 
difference  between  the  control  and  test  group  on  SAT  composite  scores,  a 
new  selection  of  groups  will  be  necessary  since  it  has  been  shown  that 
there  is  a  high  correlation  between  high  composite  SAT  and  success  in  the 
PA  course.  If  a  second  selection  process  is  necessary,  the  entire  class 
will  be  rank  ordered  on  SAT  scores.  An  equal  number  of  students  will  be 
randomly  selected  from  the  high  and  low  strata.  Sample  size  can  be 
increased  to  32  by  following  the  same  procedures  for  the  next  entering 
class,  thus  increasing  the  power  of  this  design. 

STUDENTS 

1 .  Academic  Knowledge  (Study  1) 

The  purpose  of  this  comparison  is  to  determine  whether  there  are 
differences  between  POMC  students  and  students  in  the  traditional 
curriculum  with  respect  to  academic  knowledge  of  respiratory  disease 
subject  matter.  If  the  POMC  succeeds  in  teaching  this  material  at 
least  as  well  as  currently,  then  academic  quality  will  have  been 


maintained. 
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Since  slight  differences  in  emphasis  and  content  are  found  between 
the  POMC  and  the  traditional  curriculum,  academic  test  items  from 
currently  used  block  tests  could  be  unfair  to  POMC  students.  Likewise, 
the  POMC  generated  end-of-block  test  items  could  be  biased  against  the 
traditional  students.  To  resolve  this  difficulty,  a  test  covering 
respiratory  disease  will  be  constructed  consisting  of  items  that  are 
similar  to  those  found  on  the  American  College  of  Medicine/Board 
Examination.  These  items  will  come  from  the  Educational  Testing  Service 
(ETS)  item  banks.  In  the  event  that  ETS  cannot  supply  these  items  or 
more  items  are  necessary,  experts  in  respiratory  disease  from  outside 
the  SHCS  (for  example,  University  of  Nebraska  PA  staff)  will  be  asked 
to  contribute  test  items.  These  procedures  will  insure  content  validity 
since  these  test  items  will  tap  knowledge  that  is  considered  important 
by  independent  experts  in  respiratory  disease. 

This  initial  item  pool  will  be  further  refined  by  testing  a  group 
of  Physician’s  Assistants  just  at  the  end  of  their  second  year  of  training. 
Item  statistics  will  be  computed  on  this  sample,  and  items  for  the  final 
form  of  the  test  will  be  chosen  in  accordance  with  statistical  criteria 
so  that  only  reliable  and  discriminating  items  of  appropriate  difficulty 
are  included.  In  addition,  this  procedure  will  provide  a  normative  sample 
against  which  scores  of  the  test  and  control  groups  can  be  compared. 

The  final  form  of  this  test  will  be  administered  to  test  and  control 
groups  at  the  end  of  phase  I  of  the  -ourse,  thus  delayed  retention  will 
be  measured  for  both  groups. 

2.  Problem  Solving  Skill  (Study  2) 

The  skill  to  be  assessed  in  this  comparison  is  problem  solving 
or  diagnostic  skill.  By  presenting  several  simulated  patient  encounters. 
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we  can  test  the  strident' s  ability  to  integrate  his  knowledge  toward 
successful  diagnosis  and  treatment  of  a  patient.  These  exercises  will 
essentially  be  paper  and  pencil  problem  descriptions,  but  the  student 
will  be  able  to  request  information,  prescribe  treatments,  and  note 
effects.  In  a  limited  since  he  will  be  able  to  "interact"  with  the 
simulation.  Scoring  criteria  would  include  correct  use  of  available 
information  (lab  tests,  patient  history,  physical  examinations,  special 
tests,  etc),  correct  sequence  of  steps  toward  diagnosis,  accuracy  of 
diagnosis,  appropriate  indication  of  treatment,  time  to  completion. 

Two  or  three  Simulated  Patient  Encounters  will  be  presented  to 
both  groups  at  the  end  of  the  course.  It  is  planned  that  these 
simulations  will  be  available  from  ETS  or  drawn  from  simulations  which 
are  already  in  existence.  It  is  felt  that  creation  of  simulations 
by  the  PLATO  and  PA  staffs  should  be  avoided  to  eliminate  bias. 

3.  Student  Attitude  Questionnaire  (Study  3) 

The  main  objective  of  this  study  is  to  discover  whether  POMC 
students  have  positive  (or  negative)  attitudes  toward  the  curriculum 
and/or  the  PLATO  system.  To  this  end,  a  26  item  version  of  the 
Brown  questionnaire,  used  by  Florida  State  University  in  a  recent 
study,  has  been  selected.  This  instrument  has  demonstrated  reliability 
with  college  students  (r=.80),  and  includes  many  specific  questions 
which  will  permit  inferences  about  the  various  features  of  PLATO  (e.g. 
perceived  reliability  of  terminals,  depersonalization,  etc). 

It  is  anticipated  that  the  attitude  questionnaire  will  be  given 
in  the  first  and  twelfth  month  of  the  course  in  order  to  detect 
differences  as  experience  with  the  POMC  and  PLATO  increases.  Analyses 
of  the  relationship  between  attitude  and  performance  on  the  final 
academic  test  and  simulations  are  planned. 


4.  Lesson  and  Block  Tests  (Study  4) 


Here  the  evaluation  will  be  descriptive  and  formative  in  contrast 
to  the  previous  study  which  was  comparative  and  summative.  The  goal  is 
to  show  that  POMC  students  are  meeting  learning  objectives  as  they 
progress  through  the  course.  Number  of  lesson  repeats,  number  and 
duration  of  special  remediations,  eliminations  or  washbacks  will  be 
reported . 

AUTHORS 

1 .  Performance  (Study  5) 

Author  performance  will  be  measured  in  three  ways:  (a)  average 
quality  of  lessons  produced  (as  judged  by  University  of  Illinois 
Personnel  and  student),  (b)  supervisory  ratings  of  performance,  and  (c) 
peer  ratings  of  performance.  Performance  dimensions  to  be  rated  will 
include  programming  ability  (competence  with  Tutor  language),  efficiency 
in  lesson  design,  creative  use  of  instructional  techniques,  novel 
applications  of  PLATO,  etc. 

Questionnaire  instruments  will  be  designed  by  AFHRL  for  administration 
as  soon  as  possible,  so  that  time  comparisons  can  be  made  to  show  the 
effect,  if  any,  of  experience. 

2 .  Attitudes/Opinions  (Study  6) 

Attitudes  of  authors  will  be  measured  primarily  by  a  question¬ 
naire  yet  to  be  developed.  Two  purposes  are  paramount  here — (a)  to  be 
able  to  describe  specific  strengths  and  weaknesses  of  the  PLATO  equipment, 
and  (b)  to  investigate  the  relationship  between  attitude  and  performance. 
Dimensions  to  be  sampled  include:  attitudes  toward  CAI  in  general, 
feelings  about  individualized  instruction  and  the  problem-oriented 
approach,  human  factors  aspects  of  terminal  design,  problems  in  lesson 


design,  etc. 
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Questionnaire  data  should  be  supplemented  by  interview  data  with 
interviews  to  be  conducted  by  AFH&L.  Also  open  ended  opportunities  will 
be  included  in  the  questionnaire  format. 

Two  measurements  of  attitudes  and  performance  will  be  made  at  different 
times.  Cross- lagged  correlation  analysis  may  yield  some  insight  into 
whether  attitudes  cause  performance  or  vice  versa. 

3.  Characteristics  (Study  7) 

An  attempt  will  be  made  to  relate  author  personal  and  background 
characteristics  to  performance  on  the  project.  To  this  end,  data  will  be 
collected  on  such  variables  as  level  and  t;  pe  of  education,  age,  rank, 

IQ,  attitudes,  volunteer/nonvolunteer  status,  interests,  etc.  The  aim 
here  is  to  predict  what  kinds  of  people  make  successful  authors  for 
possible  selection  purposes. 


PLATO  IV 


1.  Cost  (Study  8) 

Since  the  PLATO  Development  Branch  is  a  cost  center,  aggregate 
operating  and  development  cost  figures  can  be  readily  obtained.  Management 
analysis  personnel  will  be  contacted  for  consultation  purposes.  Capital 
Investment  figures  should  be  available  from  ARPA.  A  projection  of 
expansion  costs  under  several  scenarios  should  be  a  part  of  the  nal 
report . 

2.  Reliability  (Study  9) 

Logs  will  be  maintained  to  show  number  of  terminals  in  commission 
per  day,  percent  of  down-time  per  terminal,  cause  of  failure.  Time  phased 
analyses  can  be  made  to  detect  trends. 


SUMMARY 

Figure  2  gives  an  overview  and  summary  of  the  plan  with  responsible 
agencies  and  deadlines  specified. 


MILESTONES*  AND  RESPONSIBLE  AGENCIES** 
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appendix  d 

Proposal  for  Phase  II 
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9  December  1975 

AN  EVALUATION  OF  THE  EDUCATIONAL  AND  COST  EFFECTIVENESS  OF  THE 

PLATO  SYSTEM 


I.  Introduction 

The  CERL  proposal,  "A  Coordinated  Evaluation  of  the  Use  of  PLATO  at 
Military  Training  Sites,"  dated  17  Nov  75,  cites  the  basic  question  toward 
which  research  must  be  directed  as:  "Under  what  conditions  can  CBE  make  a  cost 
effective  contribution  to  the  instructional  needs  of  the  services." 

On  the  basis  of  the  experience  gained  through  current  research  effort  into 
the  development  of  a  new  problem  oriented  curriculum  at  the  physician  assistant 
level,  using  the  PLATO  IV  system,  the  School  of  Health  Care  Sciences  is  in  a 
unique  position  to  assist  in  research  of  a  subset  of  this  question,  specifi¬ 
cally,  "can  applications  for  the  PLATO  IV  system  and  related  instructional  pro¬ 
grams  be  found  within  a  military  training  environment  which  are  instructionally 
effective  and  cost  effective." 

The  Sheppard  capability  to  pursue  this  question  centers  on  the  fact  that 
a  trained  staff  is  already  in  place;  physical  facilities,  including  a  PLATO 
classroom  with  necessary  terminals,  carrels,  communications  equipment,  power 
and  supply  already  exist;  and  the  SHCS  offers  a  wide  variety  of  courses  designed 
to  provide  training  in  numerous  specialities  at  all  skill  levels  using  a  wide 
range  of  instructional  techniques  and  strategies.  Therefore,  the  SHCS  offers 
an  ideal  environment  in  which  the  continued  exploration  of  the  effectiveness 
of  CBE  can  be  evaluated. 

II.  Rationale 

As  noted  previously,  the  examination  of  both  the  educational  and  cost 
effectiveness  of  the  PLATO  system  in  military  training  is  a  question  that  must 
be  addressed.  The  current  research  effort  is  not  designed  to  meet  this  need  for 
a  number  of  reasons;  specifically,  it  primarily  addresses  the  impact  of  a  major 
curricular  modification  in  a  course  that  is  unique  in  military  medical  training. 
Additionally,  the  highly  specialized  nature  of  the  Physician  Assistant  course 
and  its  inherent  time  constraints  prohibit  the  demonstration  of  either  actual  or 
potential  time,  or  cost  savings. 

A  preliminary  survey  of  courses  conducted  within  the  Departments  of  Medicine, 
Dentistry,  Nursing,  Biomedical  Sciences,  and  Health  Care  Administration  revealed 
a  number  of  existing  courses  which  appear  to  be  well  suited  to  investigation  of 
the  cost  and  instructional  effectiveness  question  cited  earlier. 
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III.  Proposed  Research  Design 

The  major  issues  of  cost  and  instructional  effectiveness  can  best  be 
addressed  by  a  series  of  individual,  but  well  integrated  studies.  The  key 
to  the  success  of  any  research  in  this  area  is  the  selection  of  a  course  in 
which  optimum  potential  for  cost  and  instructional  effectiveness  is  perceived. 
Additionally,  it  must  be  understood  that  the  emphasis  of  any  research  under¬ 
taken  at  this  time  must  focus  on  evaluation  aspects  rather  than  an  extensive 
courseware  development  effort. 

To  insure  identification  of  the  optimum  course,  a  number  of  tentative 
selection  criteria  are  suggested  below.  Criteria  were  selected  to  minimize 
implementation  impact,  maximize  data  collection  potential,  and  utilize  existing 
resources.  Proposed  criteria  include: 

a.  maximum  class  size 

b.  trained  personnel  requirement  (TPR/for  year) 

c.  course  length 

d.  class  entry  schedule 

e.  instructional  methods  (lecture,  self-pacing,  etc) 

f.  PLATO  staff  backgrounds 

g .  hardware  resources 

h.  homogenity  and  interest  of  traditional  faculty 

i.  transferability  to  civilian  community 

Preliminary  application  of  these  criteria  to  the  courses  currently  being 
taught  at  the  SHCS  resulted  in  ten  potential  candidate  courses  which  were  de¬ 
serving  of  additional  evaluation  and  consideration.  These  courses  are: 


Course 

Dept 

Length 

TPR 

Max  Gp  Size 

Entry  Schedule 

Radiology 

Specialist 

MS  DM 

14  wk 

237 

24 

4 

IDT 

MS  DM 

9  wk 

32+ 

14 

13 

Diet  Therapy 

MSDB 

51  wk 

112 

12 

3 

Med  Lab  Spec 

MSDB 

17  wk 

373 

32 

3 

Op  Rm  Spec 

MS  DM 

8  wk 

228 

16 

3 
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Course 

Dept 

Length 

TPR 

Max  Gp  Size 

Entry  Schedule 

Med  Admin 
Spec 

MSDH 

6  wk 

232+ 

20 

2 

Med  Mat. 

Spec 

MSDH 

6  wk 

265 

20 

3 

Med  Rec. 

Mgmt 

MSDH 

5i  wk 

90 

15 

9 

Dent.  Spec 

MSDD 

9  wk 

255 

15 

3 

Prev.  Dent. 
Spec 

MSDD 

wk 

90 

12 

6 

IV.  Proposed  Studies 

Studies  which  can  be  accommodated  under  this  proposal  will  provide  CERL 
and  the  sponsor  valuable  data  with  which  to  address  many  of  the  questions  put 
forth  in  the  CERL  proposal.  Specific  studies  of  interest  to  ATC,  AFHRL,  ARPA, 

CERL  and  SHCS  can  be  designed  to  address  the  following  topics : 

a.  CBE  lesson  effectiveness  versus  lectures 

b.  CBE  lesson  effectiveness  versus  other  self-paced  media 

c.  effectiveness  of  alternate  instructional  strategies  in  CBE  lessons. 

d.  lesson  optimization  in  relation  to  time  savings 

e.  cost  effectiveness  determination 

f.  attitudinal  data 

(1)  students 

(2)  CBE  authors 

(3)  traditional  instructors 

In  order  to  avoid  the  evaluation  problems  which  impact  the  on-going  research 
effort,  every  effort  will  be  made  to  insure  that  comparable  cost  and  effectiveness 
data  can  be  collected  from  both  the  proposed  PLATO  application  and  the  traditional 
(control)  course. 
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V.  Resources  Required 

Personnel  -  current  authorizations  less  physician 
Equipment  -  all  required  equipment  is  currently  in  place 
Time  -  1  year 

Cost  -  Approximately  $90K  (Communications  &  Computer  Support) 


AN  ANALYSIS  OF  POMC  COURSEWARE  AT  SHEPPARD  AFB 


Eileen  Call-Himwich 


COMPUTER-BASED  EDUCATION  RESEARCH  LABORATORY 
UNIVERSITY  OF  ILLINOIS,  Urbana-Champaign 
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INTRODUCTION 


This  chapter  critiques  Sheppard's  problem-oriented  medi¬ 
cal  curriculum  courseware.  The  problem  with  reviewing  a 
group  of  lessons  together  is  that  it's  difficult  to  talk  in 
anything  but  averages  and  generalities.  Without  describing 
each  lesson  separately,  it's  hard  to  convey  the  variations  in 
approach  and  quality  in  this  potpourri  of  both  the  creative 
and  the  commonplace.  The  Sheppard  project  was  partially 
staffed  by  some  of  the  most  intelligent,  conscientious 
authors  at  any  ARPA/PLATO  site.  These  authors  produced 
inventive  lessons,  adapting  well  to  the  unique  capabilities 
and  necessities  involved  in  using  a  computer-based  approach 
to  education.  Other  authors  clung  to  old  habits,  transfer¬ 
ring  a  lecture-like  technique  to  PLATO.  Some  authors  had  no 
teaching  experience,  but  modeled  their  lessons  after  the  sort 
of  instruction  they  themselves  had  been  exposed  to  as  stu¬ 
dents.  Some  authors  welcomed  the  challenge  while  others 
resisted  the  change.  Attitudes  and  comments  ranged  from,  "I 
was  taught  by  lectures  and  struggled  through,  so  why  should 
other  students  feel  entitled  to  something  better?"  to,  "Les¬ 
son  writing  is  a  tremendously  creative,  personal  process... 
it's  like  giving  birth!" 

What  this  chapter  describes,  then,  is  the  middle  ground, 
the  level  where  most  authors  operated,  where  most  lessons 
"worked".  The  point,  after  all,  was  not  to  "be  creative",  but 
to  produce  approximately  150  student  contact  hours  of  compu¬ 
terized  instruction. 
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PROBLEM  ORIF.MTATTOM 

As  was  discussed  in  "Critical  Analysis  of  the  Initial 
Phase  of  the  Sheppard  AFB  CBF  Pro.iect-  The  Problem  Oriented 
Medical  Curriculum",  Misselt.  1978,  a  single,  official  defi¬ 
nition  of  probl em-or ientat ion  was  never  really  made.  Conse¬ 
quently,  there  were  as  many  opinions  of  the  meaning  of  pro¬ 
blem-orientation  as  there  were  authors.  Sheppard  courseware 
reflected  this  divergence  of  opinion.  Sheppard  lessons  were 
usually  one  of  three  types: 

1.  content-oriented 

2.  scenar io-oriented 

3 .  case-oriented . 

All  four  types  dispense  information,  but  with  differing 
emphases  or  focal  points.  In  the  first  case,  information  is 
presented  in  a  straightforward,  almost  lecture-like  fashion. 
The  emphasis  is  on  lesson  content  only.  These  lesson  authors 
felt  the  curriculum  would  (and  should)  be  probl em-or i ented  as 
a  result  of  other  kinds  of  lessons.  These  authors  did  not 
consider  their  content-or  iented  lessons  port,  of  the  problem- 
oriented  approach. 

Authors  using  a  "scenario"  approach  often  introduced  the 
lesson  with  a  clinical  depiction  or  situation.  For  example, 
one  lesson  opens  with  the  following  scene: 

"You're  walking  on  the  surgical  ward  and  as  you 
pass  a  room  you  hear  a  yell  and  a  noise  like  a 
person  falling  on  the  floor.  You  rush  in  and 
there  on  the  floor  is  a  patient  ('15-50  ypar  old 
male)  not  breathing,  no  heartbeat,  no  blood 
pressure  present.  You  immediately  begin  CPR 
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(cardiopulmonary  resuscitation)  and  call  for 
help,  but  in  spite  of  all  the  effort  to  re¬ 
suscitate  the  patient  he  is  pronounced  dead. 

As  you  get  up  you  notice  a  bedpan  on  the  bed. 

WHAT  HAPPENED?  "  * 

Throughout  the  remainder  of  the  lesson,  the  author  used  the 
opening  setting  as  a  springboard  to  introduce  relevant  infor¬ 
mation,  frequently  refering  back  to  the  clinical  scene.  How¬ 
ever,  the  student  is  not  expected  to  diagnose  or  "solve"  the 
case . 

Finally,  authors  sometimes  used  a  case-or iented  approach. 
The  third  trimester  was  to  be  almost  totally  comprised  of 
simulated  patient  encounters  in  which  the  student  was  expected 
to  diagnose  and  treat  the  condition.  In  addition,  some  les¬ 
sons  centered  on  less  advanced  simulations  in  which,  rather 
than  diagnosing  and  treating  the  case,  the  student  had  only 
to  assess  the  case  in  some  other  way.  For  example,  in  one 
lesson  the  student  is  presented  with  an  imaginary  dialogue 
between  a  patient  and  doctor.  The  student's  task  is  to 
determine  whether  the  information  the  patient  gives  does  or 
does  not  contribute  to  making  a  diagnosis.  If  the  patient 
information  is  indeed  important,  the  lesson  branches  to  a 
brief  discussion  of  the  implications  and  importance  of  that 
piece  of  information.  In  a  lesson  on  how  to  take  a  complete 
patient  history,  the  student  again  sees  a  patient/doctor  dia¬ 
logue,  and  must  decide  whether  the  patient  information  is 
subjective  or  objective. 


*An  aside  for  those  who  are  curious  about  the  fate  of  our 
patient.  Straining  to  use  the  bedpan  had  caused  massive  pul¬ 
monary  emboli — a  common  concern  with  postoperati ve  patients. 
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LESSON  DEVELOPMENT 

Student  Population 

Sheppard  Physician  Assistant  students  are  a  homogeneous 
group  of  highly  motivated,  hardworking  people.  All  PA  stu¬ 
dents  are  noncommissioned  officers  with  3-6  years  of  pre¬ 
vious  medical  experience.  They  must  also  have  completed  a 
Medical  Service  Fundamentals  course  and  training  at  a  tech¬ 
nical  school.  Thus,  PA  students  are  older  than  most  military 
students,  and  serious  about  medicine  as  a  career  both  in  and 
out  of  the  military.  For  this  reason,  they  are  generally 
self-motivated  and,  as  one  officer  stated,  "...study  as  long 
and  hard  as  MDs." 

Cognitive  Levels 

The  Specialty  Training  Standards  (STS  917x0  for  AFSCs 
91730/70,  23  February  1972)  for  the  Physician  Assistant 
course  prescribe  student  objectives  at  3  different  levels: 

1.  task  performance 

2.  task  knowledge 

3.  subject  knowledge. 

The  following  section  is  taken  directly  from  the  STS 
"qualitative  requirements"  for  the  Physician  Assistant 
Course.  It  lists  the  general  cognitive  levels  and  perform¬ 
ance  skills  specified  in  the  STS.  The  remainder  of  the  STS 
expands  on  all  areas  of  the  course,  enumerating  which  of 
these  skills  apply  to  what  areas.  Since  the  vast  majority 
of  the  Sheppard/ PL ATC  material  was  aimed  at  content  rather 
than  performance,  only  the  objectives  for  task  and  subject 
knowledge  have  been  included  here. 
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Task  Knowledge 

a.  Can  name  parts,  tools,  and  simple  facts  about  the  task. 
(NOMENCLATURE) 

b.  Can  determine  step  by  step  procedures  for  doing  the  task. 

( PROCEDURES  ) 

c.  Can  explain  why  and  when  the  task  must  be  done  and  why 
each  step  is  needed.  (OPERATING  PRINCIPLES) 

d.  Can  predict,  identify,  and  resolve  problems  about  the 
task.  (COMPLETE  THEORY) 

Subject  Know led  Re 

A.  Can  identify  basic  facts  and  terms  about  the  subject. 

(FACTS  ) 

B.  Can  explain  relationship  of  basic  facts  and  state  general 
principles  about  the  subject.  (PRINCIPLES) 

C.  Can  analyze  facts  and  principles  and  draw  conclusions 
about  the  subject  (ANALYSIS) 

U .  Can  evaluate  conditions  and  make  proper  decisions  about 
the  subject.  (EVALUATION) 

Sheppard  lessons  generally  emphasized  objectives  aimed 
at  nomenclature,  facts,  principles,  and  procedures.  The 
analytic,  evaluative,  and  problem  solving  levels  were  mainly 
concentrated  in  the  simulated  patient  encounters.  Though  a 
small  number  of  lessons  required  the  student  to  either  apply 
principles  to  new  situations  or  synthesize  information  and 
form  a  conclusion,  most  lessons  expected  only  that  the  student 
be  able  to  list  or  restate  information,  name  or  identify 
structural  components,  list  causes/effects,  etc.  For  example 

--State  4  factors  that  affect  the  diffusion  of  gases. 

— Name  3  methods  of  sputum  collection. 

--List  the  agents/factors  responsible  for  upper 
airway  diseases/disorders. 
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--Identify  and  name  the  bones  that  make  up  the 
thoracic  cage. 

— Name  the  4  lung  volumes  and  match  each  name  to  it's 
respective  volume. 

are  objectives  taken  from  POMC  lessons  which  typify  the  sorts 
of  tasks  and  levels  of  understanding  required  of  students. 
Certainly  some  aspects  of  clinical  science  involve" rote 
memorization,  and  often  all  that  needs  to  be  tested  is 
whether  the  student  has  actually  committed  them  to  memory. 
However,  the  problem  solving  skills  alluded  to  in  both  the 
STS  and  course  definitions  of  the  POMC  were  only  intermit¬ 
tently  reflected  in  actual  behavioral  objectives  or  stressed 
in  the  lessons  themselves.  One  possible  cause  was  that  objec¬ 
tives  were  frequently  written  after  the  lesson  itself  was 
written.  Often  if  it  was  pointed  out  that  the  objectives  and 
lesson  content  were  at  variance,  the  author  would  rewrite  the 
objectives.  Many  times  authors  seemed  not  to  have  a  clear 
idea  either  of  what  they  wanted  the  student  to  be  able  to  do 
by  the  end  of  the  lesson  or  of  what  actual  on-the-job  skills 
the  student  needed.  This  may  have  stemmed  in  part  from  the 
fact  that  few  authors  did  any  sort  of  task  analysis  before 
writing  either  the  objectives  or  the  lesson  itself.  This  was 
particularly  true  of  authors  who  lacked  teaching  experience. 

Development  Procedures 

All  authors  had  subject  matter  expertise  in  some  area  and 
wrote  lessons  concentrated  in  their  field.  Some  authors, 
however,  also  ended  up  writing  lessons  in  areas  they  knew 
little  or  nothing  about.  This  took  up  valuable  time  while 
they  had  to  research  lesson  material.  Also,  as  was  pointed 
out  in  Sheppard's  phase  one  report  (Steinkerchner ,  Deignan, 
Waters  4  DeLeo),  lack  of  subject  matter  expertise  hampers  a 
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programmer's  ability  to  make  "intelligent  choices"  about  dis¬ 
play,  feedback,  etc. 

All  curriculum  materials  were  on  PLATO.  There  was  no 
media  mixture.  When  MTC  reviewers  pointed  out  that  some  of 
the  material  was  not  well  suited  for  computerization  and 
could  be  more  economically  and  effectively  presented  using 
another  medium,  authors  insisted  that  all  material  had  to  be 
on  line.  For  example,  the  laboratory  testing  procedures  les¬ 
sons  were  presented  almost  in  the  form  of  an  on-line  handout. 
Laboratory  procedures  were  detailed  in  a  step-by-step  fash¬ 
ion.  Then  the  student  was  often  asked  something  like,  "What 
is  the  first  step  in  doing  a  gram  stain"  or  some  technical  or 
procedural  point.  Some  parts  of  these  lessons  did,  however, 
include  the  underlying  reasons  for  using  various  techniques, 
chemicals,  etc.  Thus  the  lessons  were  a  mixture  of  both 
theory  and  practice,  with  rationale  interlacing  pages  (some¬ 
times  as  many  as  3  full  frames  at  a  time)  of  step-by-step 
procedure.  When  asked  whether  as  a  Physician  Assistant  the 
student  would  ever  have  to  perform  these  tests,  the  author 
admitted  s/he  would  not.  When  the  MTC  reviewer  pointed  out 
that  it  might  then  be  more  effective  to  print  procedures  on  a 
hard-copy  hand  out  (since  they  were  so  textual)  the  author 
said  all  material  had  to  be  on-line  regardless  of  whether  it 
belonged  there.  The  question  of  whether  some  material  could 
really  be  taught  best  on-line  never  seemed  to  be  part  of  the 
lesson  development  process. 

As  was  briefly  discussed  in  the  previous  section,  the 
chronological  order  in  which  authors  wrote  their  lessons  var¬ 
ied  tremendously.  Toward  the  beginning  of  the  POMC  project 
an  MTC  member  provided  the  Sheppard  staff  with  a  copy  of 
Chanute's  lesson  development  and  documentation  plan  for 
possible  adaptation  to  the  Sheppard  project.  The  Sheppard 
staff  initially  agreed  that  such  an  "official"  policy  would 
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be  a  good  idea,  and  decided  to  institute  a  variation  of 
(Jhanute's  plan.  After  awhile,  however,  it  became  obvious 
that  no  standardized  development  procedures  were  actually 
being  followed.  Objectives  were  not  being  written,  off-line 
preplanning  was  sporadic,  etc.  After  this  was  pointed  out,  a 
period  of  about  two  months  followed  in  which  authors  consci¬ 
entiously  wrote  behavioral  objectives  before  beginning  pro¬ 
gramming  their  lessons.  For  some  authors  these  objectives 
represented  the  only  preplanning  they  did.  Other  authors 
wrote  outlines  or  portions  of  lessons  before  coding.  Finally, 
authors  fell  into  the  habit  of  writing  objectives  and  cri¬ 
terion  tests  after  the  lesson  was  coded.  In  this  way  objec¬ 
tives  were  tailored  to  fit  the  completed  lesson.  The  lack  of 
objectives  or  preplanning  in  at  least  one  instance  produced  a 
lengthy,  rambling  lesson  which  the  author  "wrote"  directly 
on-line.  Since  this  was  his  first  attempt  at  lesson  writing, 
he  had  not  yet  developed  the  repertoire  of  design  skills  nec¬ 
essary  to  extemporaneously  write  a  lesson.  However,  he 
regarded  lesson  writing  as  a  creative  process  that  was  only 
stifled  by  flowcharting,  outlining,  etc.  As  a  result  his 
first  lesson  took  nearly  a  year  to  write  and  polish,  and 
seemed  at  many  points  to  be  disconnected,  with  sections  which 
followed  consecutively  but  did  not  build  on  each  other.  The 
author's  belief  that  lesson  writing  is  a  creative  process  is 
a  sound  one,  and  experienced  authors  are  certainly  able  to 
write  a  fair  amount  of  material  on-line.  However,  just  as 
one  would  not  expect  a  jazz  artist  to  ad  lib  a  piece  of  music 
without  having  a  solid  musical  foundation  of  discipline  and 
knowledge,  one  could  not  reasonably  expect  that  a  "novice" 
author  could  ad  lib  a  lesson  without  a  solid  instructional 
foundation  of  design  and  media  experience. 
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LESSON  DESIGN 

Due  to  the  range  of  knowledge  and  skill  required  of  a 
physician  assistant,  Sheppard's  POMC  curriculum  covers  a  wide 
gamut  of  subjects  pivoting  mainly  around  anatomy,  bacterio¬ 
logy,  chemistry  and  physiology,  all  specific  to  the  respira¬ 
tory  system.  The  finished  lessons  fall  into  3  general  cate¬ 
gories — those  whose  purpose  is  to  present  information,  teach 
procedures  or  present  clinical  simulations.  In  addition, 
lessons  can  be  further  catalogued  in  terms  of  the  approaches 
authors  used  in  fulfilling  the  preceding  purposes.  Sheppard 
lessons  employed  4  main  instructional  formats: 

a.  lecture/question 

b.  drills 

c.  tutorials 

d.  simulations. 


Lecture/Question 

The  lecture  format  is  the  most  commonly  used  approach  in 
which  content  is  stressed  over  tasks  or  problem-solving.  The 
lesson  usually  unfolds  in  a  linear  fashion,  with  little 
divergence  from  the  main  path.  The  student  is  allowed  to  go 
forward  and  backward,  and  sometimes  is  able  to  branch  off 
into  a  special  help  section.  Lesson  material  is  typically 
dispensed  in  a  lecture/question  order  in  which  a  number  of 
frames  ("pages")  of  text  are  first  presented  followed  by  a 
question  or  questions  covering  what  the  student  has  just 
read . 

Drills 

A  few  lessons  use  a  drill  approach  in  which  the  main 
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emphasis  is  on  giving  the  student  practice  at  using  termino¬ 
logy,  interpreting  graphs,  identifying  various  structural 
components  (i.e.  "touch  the  sternum  in  this  drawing"),  label¬ 
ing  data  (is  this  information  subjective  or  objective?),  etc. 
Some  lessons  use  a  true  drill  approach  (in  which  the  student 
answers  a  number  of  items  per  concept),  while  other  lessons 
use  a  modified  drill  approach  (in  which  the  student  answers 
only  1  item  per  concept).  In  at  least  one  case,  namely  the 
Anatomical  Reference  Terminology  lesson,  the  material  was 
either  taught  or  gleaned  in  some  other  way,  and  the  lesson 
was  basically  a  review.  In  most  other  cases,  however,  the 
subject  matter  is  briefly  presented  at  the  beginning  of  the 
lesson,  after  which  the  student  is  drilled  on  what  s/he  has 
just  read.  For  example,  in  the  lesson  "Anatomy  of  the  Tho- 
tacic  Cage"  the  lesson  first  locates  and  labels  various  parts 
of  the  thoracic  cage.  Then  the  student  is  given  a  number  of 
drawings  of  the  thoracic  cage  on  which  s/he  must  point  out 
which  structures  are  which. 

Tutorials 

Another  approach  is  the  tutorial  in  which  the  student  is 
"walked"  through  some  procedure  step  by  step.  This  approach 
is  sometimes  used  as  a  form  of  help  after  the  student  has 
already  answered  a  question  incorrectly.  It's  also  used  in 
other  cases  such  as  the  lesson  on  spirometry  in  which  the 
student  is  led  through  a  detailed  explanation  of  lung  volumes 
and  capacities  as  demonstrated  on  a  simulated  spirograph.  In 
other  lessons,  the  student  is  guided  through  proper  proce¬ 
dures  for  taking  a  complete  patient  history,  doing  specific 
laboratory  tests,  etc. 
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Simulations 

The  most  obvious  use  of  simulations  in  the  Sheppard  cur¬ 
riculum  is  the  simulated  patient  encounters  in  which  the  stu¬ 
dent  must  collect  a  patient  history,  gather  physical  examina¬ 
tion  and  laboratory  data,  then  diagnose  and  treat  the  con¬ 
dition.  Mo  new  subject  matter  is  presented,  the  emphasis 
being  onlv  on  having  the  student  "solve"  the  case.  Fince  the 
project  ended  before  the  third  trimester  for  which  the  clini¬ 
cal  simulations  were  scheduled,  few  students  ever  saw  the 
simulations  and  the  lessons  were  not  in  polished  form  at  the 
end  of  the  project. 

A  few  other  Sheppard  lessons,  particularly  the  lessons  on 
gas  laws,  incorporated  a  different  sort  of  simulation.  Some 
of  these  lessons  tried  to  demonstrate  various  experiments  to 
illustrate  cause  and  effect  relationships.  For  example,  one 
lesson  simulates  the  effect  on  a  manometer  of  increases  and 
decreases  in  air  pressure. 
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INSTRUCTIONAL  APPROACHES 

The  following  sections  discuss  various  instructional 
approaches  employed  by  Sheppard  authors.  Lessons  are 
described  in  terms  of  the  three  main  areas  of  testing,  inter¬ 
action,  and  style. 

Testing 

All  the  POMC  lessons  have  end-of-lesson  tests.  There  is 
generally  one  item  p*>r  concept  or  student  objective.  Test 
items  are  almost  always  multiple  choice,  and,  like  the  prac¬ 
tice  items  in  the  lessons  themselves,  mainly  test  recall. 
Because  of  the  small  number  of  items,  test  reliability  (and 
hence  validity)  is  often  low.  It  is  our  understanding  that 
these  tests  were  intended  to  be  norm-referenced  rather  than 
criterion  referenced  in  nature.  Our  analyses  of  the  item 
response  data  preserved  on  the  PLATO  system  have  verified 
that  many  of  the  items  have  difficulty  and  discrimination 
indices  which  fall  outside  the  ranges  usually  recommended  for 
norm-referenced  tests.  Such  items  detract  from  the  useful¬ 
ness  of  the  tests  and  should  probably  have  been  modified  or 
eliminated  in  the  test  development  process.  Eecause  of  the 
extremely  small  sample  (n=16)  who  were  exposed  to  the  PLATO 
instruction,  however,  the  tests  were  never  empirically  vali¬ 
dated.  Hence,  the  only  data  that  were  ever  collected  for 
these  tests  were  penerated  in  actual  operational  use  of  the 
lessons.  Oiven  the  hurried  and  limited  nature  of  the  test 
development,  it  is  not  surprising  that  so  many  of  the  items 
appear  to  be  of  little  value. 

Most  authors  did  not  believe  that  there  was  any  need  or 
place  for  mastery  learning  in  the  PCMC .  While  some  authors 
thought  the  criterion  test  should  be  discriminating,  others 
thought  that  PLATO  lessons  could  only  measure  short-term 
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retention,  so  there  was  no  need  or  reason  to  measure  mastery. 
This  attitude  may  well  have  been  a  contributing  factor  to  the 
1:1  ratio  of  questions  to  objectives  found  in  most  of  the 
criterion  tests. 

Interaction 

Over  the  course  of  the  Sheppard  POMC  project,  the  amount 
of  student  interaction  (in  the  form  of  questions)  increased 
markedly.  Even  lecture-type  lessons  became  more  interspersed 
with  questions.  However,  the  majority  of  questions  were 
restricted  to  virtual  verbatim,  copy-frame  type  questions  in 
which  the  student  had  only  to  reiterate  some  bit  of  termino¬ 
logy,  list  information  or  repeat  data  usually  presented  from 
one  to  three  frames  *  before  the  question  itself.  In  one 
lesson,  for  example,  this  definition  is  given  for  upper  air¬ 
way  diseases: 


"UPPER  AIRWAY  DISEASES 

is  the  collective  term  given  to  diseases  and 
disorders  resulting  from  structural  alterations 
or  dysfunctions  of  the: 

1 .  nose 

2.  paranasal  sinuses 
3 • pharynx 

.  larynx  . " 


#A  frame  is  any  screen  display  only  initiated  by  student 
input . 
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Two  frames  later,  the  student  is  asked: 


"Upper  Airway  Diseases  have  been  defined 
as  disorders  or  dysfunctions  of  the: 

(Please  list  the  terms  one-at-a-time .  ) " 

1 . 


2. 


3. 


4 . 

In  another  lesson  on  pulmonary  emboli,  one  frame  states: 

"In  experiments,  75%  of  the  pulmonary  vascular  bed 
may  be  gradually  obstructed  before  pulmonary  hyper¬ 
tension  develops." 

On  the  page  immediately  following,  the  student  is  asked: 

"What  %  of  vascular  bed  may  be  obstructed  before 
hypertension  occurs?" 

Though  most  lesson  questions  (and  criterion  test  items) 
are  of  a  verbatim  recall  type,  a  small  number  of  the  intro¬ 
ductory  basic  science  lessons  present  the  student  with  chal¬ 
lenging  word  problem  type  questions.  In  a  lesson  on  gas  laws 
for  instance,  the  student  must  apply  Boyle’s  Law  to  solve  at 
least  4  randomly  chosen  word  problems  of  the  following  sort: 
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"An  11  liter  container  of  gas  is  initially  at  a  pre¬ 
ssure  of  14  atm.  What  will  the  pressure  be  if  the  vol¬ 
ume  is  changed  to  19  liters  at  constant  temperature?" 

Feedback  and  help .  In  the  majority  of  lessons,  feed¬ 
back  for  incorrect  student  responses  consists  of  a  general 
hint.  If  the  student  misses  the  question  a  second  time  s/he 
is  then  told  the  correct  answer  and  told  to  type  in  the  cor¬ 
rect  response.  The  general  help  may  be  one  of  a  number  of 
sorts.  One  type  is  the  form  cue  in  which,  rather  than  giving 
a  cognitive  clue,  the  hint  consists  of  the  starting  letter, 
number  of  words  in  the  response,  etc.  For  example,  in  one 
lesson  the  student  must  answer  the  following  question: 

"If  a  nasal  smear  shows  a  large  number  of  neutro- 
philes  and  eosinophils  your  patient  most  probably 
has  an  _  with  a  superimposed  _ _ ." 

If  the  student  gives  an  incorrect  response  for  the  first 
blank,  the  feedback  s/he  gets  is: 

"Two  words" 

If  the  student  answers  incorrectly  again,  the  next  hint  is: 
"Initials  ’a-r"' 

Finally,  after  the  3rd  try  the  student  is  told: 

"Answer  -  allergic  rhinitis" 

More  often  the  feedback  gives  a  content  cue  as  in  the 
following  example.  The  student  is  given  the  question: 
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"In  which  state  of  matter  are  the  molecules 
moving  MOST  rapidly?" 

If  the  student  answers  incorrectly,  the  computer's  response 
i  s : 


"Molecular  motion  in  solids  and  liquids  is  limited 

by  the  close  contact  of  neighboring  molecules." 

A  small  number  of  lessons  include  questions  with  no  help 
or  feedback  whatsoever.  An  incorrect  response  generally  eli¬ 
cits  only  a  noncommittal  "Try  again",  "Press  -BACK-  for  help" 
or  a  simple  "no".  The  question  must  be  answered  before  the 
student  is  allowed  to  continue,  hut  s/he  is  left  more  or  less 
to  her/his  own  devices  in  divining  the  answer.  While  this  is 
not  the  norm,  it  happens  with  enough  frequency  to  deserve 
comment . 

While  many  lessons  offer  the  student  "help"  at  a  ques¬ 
tion,  the  help  usually  consists  of  having  the  student  press  a 
key  which  sends  her/him  back  to  reread  some  text  s/he  has 
already  been  through.  This  seems  to  be  primarily  a  function 
of  what  cognitive  level  the  question  is  aimed  at.  Questions 
aimed  strictly  at  a  recall  level  require  little  or  no  expla¬ 
nation  (these  are  also  the  sorts  of  questions  which  have  form 
cues  for  feedback).  Questions  that  delve  into  higher  cogni¬ 
tive  levels,  however,  in  which  the  student  must  apply  rather 
than  recite  information,  necessitate  more  sophisticated  feed¬ 
back  and  help.  For  instance,  if  the  student  has  difficulty 
making  a  diagnosis  in  the  clinical  simulations,  s/he  is 
routed  to  a  socratic  sort  of  tutorial  in  which  the  student  is 
led  through  the  pertinent  history  and  physical  examination 
data  to  a  diagnosis. 
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Style 

Format ■  POMC  lessons  generally  utilize  one  of  three  sim¬ 
ilar  formats.  The  basic  structure  is  a  linear  oresentation  in 
which  al]  students  take  the  same  straight-lined  oath  through 
the  lesson.  These  lessons  sometimes  include  a  "roadrrap"  or 
"index"  which  maps  out  the  route  the  student  will  take,  but 
neither  allows  the  student  to  choose  her/his  own  path.  A 
second  structure,  a  variation  on  the  linear  format,  is  a 
design  in  which  the  student  ping-pongs  back  and  forth  between 
sections  of  the  lesson  and  a  choice-index  from  which  the  stu¬ 
dent  chooses  the  section  s/he  wants  to  study,  hut  must  take 
all  sections  in  the  order  in  which  they  appear  in  the  index. 
If  s/he  deviates  from  that  itinerary,  s/he's  told,  "Since 
this  is  the  first  time  you’ve  been  through  this  lesson,  and 
since  you  have  not  completed  section  x,  you  may  not  yet  go  to 
section  y."  An  advantage  to  this  approach  is  that  the  student 
may,  however,  choose  to  review  any  parts  of  the  lesson  s/he 
has  already  seen.  A  third  approach  is  a  completely  autonomous 
situation  in  which  the  student  sees  an  index  from  which  s/he 
can  choose  any  section  in  the  lesson  in  any  order  s/he 
wishes . 

Graphics.  The  graphics  in  POMC  lessons  also  generally 
fall  into  three  main  categories.  The  first  type  is  often  an 
animation  (as  opposed  to  a  static  illustration)  which  either 
demonstrates  or  depicts  whatever  concept  is  being  discussed. 
One  lesson,  for  instance,  has  an  animation  of  a  manometer 
which  simulates  the  effect  of  changes  in  pressure.  In  another 
lesson  the  author  plots  various  spirometrical ly  determined 
respiratory  pressures  on  a  graph  depicting  the  relative 
volume  of  each  pressure.  Graphics  of  this  sort  enhance  or  in 
some  way  broaden  the  student's  understanding  of  a  situation 
or  concept  either  by  simulating  it  or  depicting  it  from  a 
different  perspective. 
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The  second  category  consists  of  static  "pictures"  which 
mainly  serve  to  illustrate  the  text.  While  some  of  these 
drawings  are  interesting  or  even  entertaining,  they  don't 
generally  add  to  the  student's  understanding  or  bank  of  know¬ 
ledge  (except  in  the  sense  that  they  "show"  the  student  what 
a  thing  looks  like).  An  example  is  a  laboratory  procedure 
lesson  which  includes  a  drawing  of  a  laboratory  slide  with  a 
drop  of  liquid.  While  the  drawing  is  of  some  interest  it 
probably  adds  very  little  to  what  the  student  already  knows. 
This  is  not  necessarily  a  criticism.  A  textual  lesson  with 
drawings  is  more  interesting  than  one  without,  just  as  a  book 
with  pictures  is  more  interesting  than  one  without.  However, 
removing  t  e  graphics  would  probably  do  little  harm  in  terms 
of  the  student's  grasp  of  the  subject. 

Another  category  is  the  medical  illustration.  Sheppard 
had  an  extremely  talented  medical  artist  on  staff  who  created 
beautiful  anatomical  drawings.  These  drawings  made  an  impor¬ 
tant  contribution  to  the  quality  of  a  number  of  anatomy  les¬ 
sons. 

The  final  type  of  POMC  graphic  is  the  "cartoon"  type 
which  can  be  best  characterized  as  a  non-realistic  or  imagin¬ 
ary  illustration  which  students  generally  regard  as  "cute". 

An  example  is  an  imaginary  flower-like  drawing  named  A! 
Allergen  in  an  allergy  lesson  which  waves  a  petal  "hello"  at 
the  student.  Another  instance  from  the  same  allergy  series  is 
a  set  of  three  figures  used  in  a  game  in  which  the  student 
must  tell  whether  certain  foods  produce  allergies  or  not.  One 
figure  is  a  running,  man,  another  is  a  sleeping  man,  and  the 
third  is  a  smiley-faced  sun.  While  these  sorts  of  drawings 
can  make  a  frame  interesting,  if  carried  to  an  extreme,  they 
can  also  annoy  the  student.  POMC  students  commonly  referred 
to  one  series  of  such  lessons  as  "Sesame  Street",  and  felt 
they  were  a  bit  juvenile. 


